Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250110 02:51:04.988870 416 test_util.cc:274] Using random seed: 712379550
2025-01-10T02:51:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T02:51:05Z Disabled control of system clock
I20250110 02:51:05.018802 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35975
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250110 02:51:05.321671 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:05.321928 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:05.322001 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:05.329417 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 02:51:05.329527 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:05.329591 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 02:51:05.329640 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 02:51:05.337646 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:05.339051 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:05.340579 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:05.351102 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:05.352095 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:05.352949 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:05.353857 424 server_base.cc:1034] running on GCE node
I20250110 02:51:05.354575 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:05.355841 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:05.357051 424 hybrid_clock.cc:648] HybridClock initialized: now 1736477465356954 us; error 64 us; skew 500 ppm
I20250110 02:51:05.357338 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:05.359447 424 webserver.cc:458] Webserver started at http://127.0.104.62:34085/ using document root <none> and password file <none>
I20250110 02:51:05.360122 424 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:05.360692 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:05.360975 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:05.363411 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data/instance:
uuid: "8fca9a92f47145fcaaf4f65038d6f558"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.364079 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal/instance:
uuid: "8fca9a92f47145fcaaf4f65038d6f558"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.368749 424 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250110 02:51:05.371737 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:05.372846 424 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:05.372984 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal
uuid: "8fca9a92f47145fcaaf4f65038d6f558"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.373142 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:05.400796 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:05.401595 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:05.401852 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:05.421298 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35975
I20250110 02:51:05.421329 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35975 every 8 connection(s)
I20250110 02:51:05.422793 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/data/info.pb
I20250110 02:51:05.426306 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:05.431463 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:05.432690 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 424
I20250110 02:51:05.432988 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-0/wal/instance
I20250110 02:51:05.436482 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:44449
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 02:51:05.446718 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } has no permanent_uuid. Determining permanent_uuid...
W20250110 02:51:05.447780 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
W20250110 02:51:05.449141 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:44449: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
I20250110 02:51:05.498473 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } attempt: 1
W20250110 02:51:05.500123 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:44449: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
I20250110 02:51:05.568356 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } attempt: 2
W20250110 02:51:05.570055 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:44449: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
I20250110 02:51:05.661266 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } attempt: 3
W20250110 02:51:05.662881 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:44449: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
W20250110 02:51:05.742354 494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:05.742651 494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:05.742775 494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:05.749962 494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 02:51:05.750098 494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:05.750165 494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 02:51:05.750240 494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 02:51:05.758292 494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:05.759835 494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:05.761804 494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:05.771417 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:05.772257 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:05.773038 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:05.773911 494 server_base.cc:1034] running on GCE node
I20250110 02:51:05.774641 494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:05.775871 494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:05.777096 494 hybrid_clock.cc:648] HybridClock initialized: now 1736477465777037 us; error 42 us; skew 500 ppm
I20250110 02:51:05.777405 494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:05.779510 494 webserver.cc:458] Webserver started at http://127.0.104.61:46707/ using document root <none> and password file <none>
I20250110 02:51:05.780186 494 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:05.780293 494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:05.780546 494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:05.783035 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/instance:
uuid: "8bc934cc708045e9858ee8f8e64660cd"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.783737 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal/instance:
uuid: "8bc934cc708045e9858ee8f8e64660cd"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.788895 494 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 02:51:05.791842 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:05.793087 494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:05.793241 494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
uuid: "8bc934cc708045e9858ee8f8e64660cd"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:05.793412 494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:05.806090 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } attempt: 4
W20250110 02:51:05.807719 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:44449: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
I20250110 02:51:05.814654 494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:05.815325 494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:05.815546 494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:05.831468 494 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:44449
I20250110 02:51:05.831485 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:44449 every 8 connection(s)
I20250110 02:51:05.832593 494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
I20250110 02:51:05.835966 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:05.840636 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 494
I20250110 02:51:05.840768 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal/instance
I20250110 02:51:05.841773 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39025
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 02:51:05.841655 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:05.854003 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:05.862260 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
W20250110 02:51:05.863318 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
W20250110 02:51:05.864542 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:05.913913 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 1
W20250110 02:51:05.915535 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:05.983731 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 2
W20250110 02:51:05.985601 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:06.076835 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 3
W20250110 02:51:06.078714 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:06.106927 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } attempt: 5
I20250110 02:51:06.117087 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
W20250110 02:51:06.118865 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
W20250110 02:51:06.155236 564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:06.155475 564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:06.155543 564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:06.162278 564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 02:51:06.162408 564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:06.162487 564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 02:51:06.162588 564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 02:51:06.170092 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 1
W20250110 02:51:06.171710 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:06.172259 564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:06.173628 564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:06.175233 564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:06.186512 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:06.187394 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:06.187948 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:06.188853 564 server_base.cc:1034] running on GCE node
I20250110 02:51:06.189534 564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:06.190871 564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:06.192072 564 hybrid_clock.cc:648] HybridClock initialized: now 1736477466192026 us; error 24 us; skew 500 ppm
I20250110 02:51:06.192374 564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:06.194428 564 webserver.cc:458] Webserver started at http://127.0.104.60:34089/ using document root <none> and password file <none>
I20250110 02:51:06.195088 564 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:06.195202 564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:06.195466 564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:06.198050 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/instance:
uuid: "5d8d40556dc348e8aa8255d116251a10"
format_stamp: "Formatted at 2025-01-10 02:51:06 on dist-test-slave-bpn1"
I20250110 02:51:06.198800 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal/instance:
uuid: "5d8d40556dc348e8aa8255d116251a10"
format_stamp: "Formatted at 2025-01-10 02:51:06 on dist-test-slave-bpn1"
I20250110 02:51:06.203454 564 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250110 02:51:06.206580 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:06.207746 564 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250110 02:51:06.207882 564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
uuid: "5d8d40556dc348e8aa8255d116251a10"
format_stamp: "Formatted at 2025-01-10 02:51:06 on dist-test-slave-bpn1"
I20250110 02:51:06.208035 564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:06.221899 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 4
W20250110 02:51:06.223448 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:06.232234 564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:06.233441 564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:06.233704 564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:06.239917 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 2
W20250110 02:51:06.241835 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111)
I20250110 02:51:06.251132 564 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:39025
I20250110 02:51:06.251159 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:39025 every 8 connection(s)
I20250110 02:51:06.252341 564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
I20250110 02:51:06.255800 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:06.257576 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 564
I20250110 02:51:06.258294 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal/instance
I20250110 02:51:06.262984 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:06.285015 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:06.291889 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
I20250110 02:51:06.301425 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Bootstrap starting.
I20250110 02:51:06.304527 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:06.305438 632 log.cc:826] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:06.308125 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: No bootstrap required, opened a new log
I20250110 02:51:06.313647 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.313894 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:06.313977 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8d40556dc348e8aa8255d116251a10, State: Initialized, Role: FOLLOWER
I20250110 02:51:06.314463 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.315143 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.315469 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.316357 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: configured and running, proceeding with master startup.
W20250110 02:51:06.325094 652 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 02:51:06.325204 652 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 02:51:06.322733 564 master_runner.cc:186] Error getting master registration for 127.0.104.62:35975: OK, instance_id { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" instance_seqno: 1736477465415112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 02:51:06.329216 564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250110 02:51:06.348039 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 3
I20250110 02:51:06.358772 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: Bootstrap starting.
I20250110 02:51:06.362612 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:06.363868 490 log.cc:826] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:06.366670 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: No bootstrap required, opened a new log
I20250110 02:51:06.372391 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.372658 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:06.372776 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fca9a92f47145fcaaf4f65038d6f558, State: Initialized, Role: FOLLOWER
I20250110 02:51:06.373334 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.374193 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.374580 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.375126 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: configured and running, proceeding with master startup.
W20250110 02:51:06.386003 666 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 02:51:06.386127 666 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 02:51:06.383855 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:44449: OK, instance_id { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" instance_seqno: 1736477465827024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 02:51:06.390434 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 02:51:06.404136 444 tablet.cc:2367] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 02:51:06.475698 655 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:06.476033 655 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.477494 655 leader_election.cc:290] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:06.478600 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:06.479152 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fca9a92f47145fcaaf4f65038d6f558 in term 0.
I20250110 02:51:06.479779 439 leader_election.cc:304] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters:
I20250110 02:51:06.480131 655 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:06.480262 655 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:06.480363 655 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:06.482661 655 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.483794 655 leader_election.cc:290] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 1 election: Requested vote from peers 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:06.484503 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10"
I20250110 02:51:06.484697 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:06.485512 586 tablet.cc:2367] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 02:51:06.488158 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fca9a92f47145fcaaf4f65038d6f558 in term 1.
I20250110 02:51:06.489869 439 leader_election.cc:304] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters:
I20250110 02:51:06.490255 655 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:06.491158 655 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 LEADER]: Becoming Leader. State: Replica: 8fca9a92f47145fcaaf4f65038d6f558, State: Running, Role: LEADER
I20250110 02:51:06.491864 655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.502825 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fca9a92f47145fcaaf4f65038d6f558. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.503168 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:06.509997 670 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 02:51:06.513664 670 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 02:51:06.522658 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } attempt: 5
I20250110 02:51:06.523125 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Refusing update from remote peer 8fca9a92f47145fcaaf4f65038d6f558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:06.525185 668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:06.541819 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Bootstrap starting.
I20250110 02:51:06.545670 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:06.547605 561 log.cc:826] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:06.557488 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: No bootstrap required, opened a new log
I20250110 02:51:06.559331 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fca9a92f47145fcaaf4f65038d6f558. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.559614 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.563344 671 mvcc.cc:204] Tried to move back new op lower bound from 7112611702869671936 to 7112611702755278848. Current Snapshot: MvccSnapshot[applied={T|T < 7112611702869671936}]
I20250110 02:51:06.571684 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.575254 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:06.575570 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc934cc708045e9858ee8f8e64660cd, State: Initialized, Role: FOLLOWER
I20250110 02:51:06.576002 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.576287 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:06.576913 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.577174 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.578182 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:06.579013 670 catalog_manager.cc:1348] Generated new cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:06.579108 670 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 02:51:06.579633 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.579910 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.594511 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: configured and running, proceeding with master startup.
I20250110 02:51:06.594555 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd" is_pre_election: true
I20250110 02:51:06.595111 536 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fca9a92f47145fcaaf4f65038d6f558 in term 0.
I20250110 02:51:06.595463 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd"
I20250110 02:51:06.595731 535 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:06.599411 535 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fca9a92f47145fcaaf4f65038d6f558 in term 1.
I20250110 02:51:06.599434 670 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 02:51:06.600245 534 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Refusing update from remote peer 8fca9a92f47145fcaaf4f65038d6f558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:06.601372 670 catalog_manager.cc:1505] Loading token signing keys...
I20250110 02:51:06.601310 655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:06.622560 670 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: Generated new TSK 0
I20250110 02:51:06.623493 670 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 02:51:06.636538 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 02:51:06.645947 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.646332 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.646704 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fca9a92f47145fcaaf4f65038d6f558. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.646931 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:06.647347 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fca9a92f47145fcaaf4f65038d6f558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:06.647681 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:06.694693 694 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250110 02:51:07.034969 690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:07.035230 690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:07.035319 690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:07.042094 690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:07.042268 690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 02:51:07.050563 690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:07.052001 690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:07.053529 690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:07.065284 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:07.065829 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:07.066730 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:07.067562 690 server_base.cc:1034] running on GCE node
I20250110 02:51:07.068089 690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:07.068905 690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:07.070098 690 hybrid_clock.cc:648] HybridClock initialized: now 1736477467070044 us; error 36 us; skew 500 ppm
I20250110 02:51:07.070351 690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:07.072377 690 webserver.cc:458] Webserver started at http://127.0.104.1:42251/ using document root <none> and password file <none>
I20250110 02:51:07.073015 690 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:07.073123 690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:07.073382 690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:07.075695 690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data/instance:
uuid: "98336b684b41407da8a1949e30cd82e0"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.076457 690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/instance:
uuid: "98336b684b41407da8a1949e30cd82e0"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.081455 690 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 02:51:07.084391 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.085994 690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:07.086161 690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal
uuid: "98336b684b41407da8a1949e30cd82e0"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.086318 690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:07.101029 690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:07.101737 690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:07.102008 690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:07.102867 690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 02:51:07.104117 690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 02:51:07.104210 690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.104323 690 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 02:51:07.104374 690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.124080 778 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fca9a92f47145fcaaf4f65038d6f558)
I20250110 02:51:07.124238 778 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.125373 778 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
I20250110 02:51:07.126044 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd" is_pre_election: true
I20250110 02:51:07.126003 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:07.126917 584 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10; no voters: 8bc934cc708045e9858ee8f8e64660cd, 8fca9a92f47145fcaaf4f65038d6f558
I20250110 02:51:07.127336 778 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:07.139313 690 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45139
I20250110 02:51:07.139348 821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45139 every 8 connection(s)
I20250110 02:51:07.140678 690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/data/info.pb
I20250110 02:51:07.143718 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 690
I20250110 02:51:07.143915 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/instance
I20250110 02:51:07.150770 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 02:51:07.184302 834 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fca9a92f47145fcaaf4f65038d6f558)
I20250110 02:51:07.184574 834 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.186834 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:07.187139 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc934cc708045e9858ee8f8e64660cd in term 1.
I20250110 02:51:07.187597 510 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8bc934cc708045e9858ee8f8e64660cd; no voters:
I20250110 02:51:07.189426 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:07.193634 834 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:07.193804 834 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 02:51:07.194264 834 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fca9a92f47145fcaaf4f65038d6f558)
I20250110 02:51:07.195274 834 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 1 FOLLOWER]: Advancing to term 2
I20250110 02:51:07.195559 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:07.195880 824 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.196821 824 heartbeater.cc:510] Master 127.0.104.61:44449 requested a full tablet report, sending...
I20250110 02:51:07.198103 834 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.198969 834 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 2 election: Requested vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:07.199287 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558"
I20250110 02:51:07.199556 525 ts_manager.cc:194] Registered new tserver with Master: 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:07.202437 823 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35975
I20250110 02:51:07.202472 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:07.202665 823 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.202662 822 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.203341 822 heartbeater.cc:510] Master 127.0.104.60:39025 requested a full tablet report, sending...
I20250110 02:51:07.203358 823 heartbeater.cc:510] Master 127.0.104.62:35975 requested a full tablet report, sending...
I20250110 02:51:07.205067 597 ts_manager.cc:194] Registered new tserver with Master: 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:07.206086 454 ts_manager.cc:194] Registered new tserver with Master: 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:07.207571 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:58735
I20250110 02:51:07.213092 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10"
I20250110 02:51:07.213549 510 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8bc934cc708045e9858ee8f8e64660cd; no voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558
I20250110 02:51:07.213840 836 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250110 02:51:07.327694 652 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:07.328301 652 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: loading cluster ID for follower catalog manager: success
I20250110 02:51:07.331490 652 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: acquiring CA information for follower catalog manager: success
I20250110 02:51:07.333513 652 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250110 02:51:07.510706 833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:07.510950 833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:07.511050 833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:07.518090 833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:07.518301 833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 02:51:07.527226 833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:07.528661 833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:07.530146 833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:07.541940 842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:07.542773 844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:07.543336 841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:07.544157 833 server_base.cc:1034] running on GCE node
I20250110 02:51:07.544728 833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:07.545534 833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:07.546746 833 hybrid_clock.cc:648] HybridClock initialized: now 1736477467546689 us; error 37 us; skew 500 ppm
I20250110 02:51:07.547006 833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:07.549124 833 webserver.cc:458] Webserver started at http://127.0.104.2:35155/ using document root <none> and password file <none>
I20250110 02:51:07.549782 833 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:07.549896 833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:07.550170 833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:07.552524 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data/instance:
uuid: "85f9597077934f1a9814d284ed06d13d"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.553181 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/instance:
uuid: "85f9597077934f1a9814d284ed06d13d"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.558240 833 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250110 02:51:07.561230 850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.562422 833 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 02:51:07.562561 833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal
uuid: "85f9597077934f1a9814d284ed06d13d"
format_stamp: "Formatted at 2025-01-10 02:51:07 on dist-test-slave-bpn1"
I20250110 02:51:07.562716 833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:07.586716 833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:07.587438 833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:07.587690 833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:07.588487 833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 02:51:07.589975 833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 02:51:07.590049 833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.590168 833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 02:51:07.590220 833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:07.616668 535 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Rejecting Update request from peer 8fca9a92f47145fcaaf4f65038d6f558 for earlier term 1. Current term is 2. Ops: []
I20250110 02:51:07.617405 835 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 02:51:07.617780 835 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 02:51:07.617918 835 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8fca9a92f47145fcaaf4f65038d6f558, State: Running, Role: LEADER
I20250110 02:51:07.618253 835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.618835 835 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 1 FOLLOWER]: Advancing to term 2
I20250110 02:51:07.624099 833 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33221
I20250110 02:51:07.624131 962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33221 every 8 connection(s)
I20250110 02:51:07.625253 833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/data/info.pb
I20250110 02:51:07.630532 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 833
I20250110 02:51:07.630769 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/instance
I20250110 02:51:07.646857 963 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:07.646857 964 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35975
I20250110 02:51:07.647107 963 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.647110 964 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.647387 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:07.647522 965 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:07.648134 965 heartbeater.cc:510] Master 127.0.104.61:44449 requested a full tablet report, sending...
I20250110 02:51:07.648168 963 heartbeater.cc:510] Master 127.0.104.60:39025 requested a full tablet report, sending...
I20250110 02:51:07.648203 964 heartbeater.cc:510] Master 127.0.104.62:35975 requested a full tablet report, sending...
I20250110 02:51:07.649518 454 ts_manager.cc:194] Registered new tserver with Master: 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:07.649555 597 ts_manager.cc:194] Registered new tserver with Master: 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:07.650132 525 ts_manager.cc:194] Registered new tserver with Master: 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:07.699048 694 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:07.699160 694 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: loading cluster ID for follower catalog manager: success
I20250110 02:51:07.701728 694 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: acquiring CA information for follower catalog manager: success
I20250110 02:51:07.704294 694 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 02:51:07.704435 971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8fca9a92f47145fcaaf4f65038d6f558)
I20250110 02:51:07.704568 971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.705070 971 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
I20250110 02:51:07.705427 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd" is_pre_election: true
I20250110 02:51:07.705425 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:07.705703 535 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d8d40556dc348e8aa8255d116251a10 in current term 2: Already voted for candidate 8bc934cc708045e9858ee8f8e64660cd in this term.
I20250110 02:51:07.705667 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d8d40556dc348e8aa8255d116251a10 in term 2.
I20250110 02:51:07.706164 583 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters: 8bc934cc708045e9858ee8f8e64660cd
I20250110 02:51:07.706454 971 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 02:51:07.706579 971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8fca9a92f47145fcaaf4f65038d6f558)
I20250110 02:51:07.706676 971 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 1 FOLLOWER]: Advancing to term 2
I20250110 02:51:07.709635 823 heartbeater.cc:502] Master 127.0.104.62:35975 was elected leader, sending a full tablet report...
I20250110 02:51:07.709822 971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.710453 971 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 election: Requested vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
I20250110 02:51:07.711184 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558"
I20250110 02:51:07.711185 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd"
I20250110 02:51:07.711408 535 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d8d40556dc348e8aa8255d116251a10 in current term 2: Already voted for candidate 8bc934cc708045e9858ee8f8e64660cd in this term.
I20250110 02:51:07.713104 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d8d40556dc348e8aa8255d116251a10 in term 2.
I20250110 02:51:07.713402 583 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters: 8bc934cc708045e9858ee8f8e64660cd
I20250110 02:51:07.713624 971 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 02:51:07.713972 971 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 LEADER]: Becoming Leader. State: Replica: 5d8d40556dc348e8aa8255d116251a10, State: Running, Role: LEADER
I20250110 02:51:07.714617 971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.716259 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8d40556dc348e8aa8255d116251a10. Latest consensus state: current_term: 2 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:07.716754 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:07.723999 974 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 02:51:07.725371 974 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 02:51:07.726102 974 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:07.726183 974 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 02:51:07.727217 974 catalog_manager.cc:1505] Loading token signing keys...
I20250110 02:51:07.727851 974 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Loaded TSK: 0
I20250110 02:51:07.728572 974 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 02:51:07.732460 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 02:51:07.741394 976 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:07.741631 976 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.742583 976 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:07.743070 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:07.743192 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:07.743323 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc934cc708045e9858ee8f8e64660cd in term 2.
I20250110 02:51:07.743774 512 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc934cc708045e9858ee8f8e64660cd, 8fca9a92f47145fcaaf4f65038d6f558; no voters:
I20250110 02:51:07.744089 976 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 02:51:07.744220 976 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:07.744318 976 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 2 FOLLOWER]: Advancing to term 3
I20250110 02:51:07.747483 976 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.748396 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558"
I20250110 02:51:07.748441 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10"
I20250110 02:51:07.748582 465 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 2 FOLLOWER]: Advancing to term 3
I20250110 02:51:07.748948 976 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 3 election: Requested vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:07.751040 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc934cc708045e9858ee8f8e64660cd in term 3.
I20250110 02:51:07.751415 512 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8bc934cc708045e9858ee8f8e64660cd, 8fca9a92f47145fcaaf4f65038d6f558; no voters: 5d8d40556dc348e8aa8255d116251a10
I20250110 02:51:07.751718 976 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 FOLLOWER]: Leader election won for term 3
I20250110 02:51:07.752156 976 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 LEADER]: Becoming Leader. State: Replica: 8bc934cc708045e9858ee8f8e64660cd, State: Running, Role: LEADER
I20250110 02:51:07.752700 976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:07.754532 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc934cc708045e9858ee8f8e64660cd. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:07.754886 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:07.755510 979 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 02:51:07.757966 979 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 02:51:07.758935 979 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:07.759022 979 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 02:51:07.760476 979 catalog_manager.cc:1505] Loading token signing keys...
I20250110 02:51:07.761425 979 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Loaded TSK: 0
I20250110 02:51:07.762413 979 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250110 02:51:08.062189 975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:08.062484 975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:08.062636 975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 02:51:08.069973 975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:08.070194 975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 02:51:08.078684 975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:08.080175 975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:08.081702 975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 02:51:08.093318 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:08.094249 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:08.095001 987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:08.095844 975 server_base.cc:1034] running on GCE node
I20250110 02:51:08.096956 975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:08.097827 975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:08.099027 975 hybrid_clock.cc:648] HybridClock initialized: now 1736477468098973 us; error 31 us; skew 500 ppm
I20250110 02:51:08.099292 975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:08.101394 975 webserver.cc:458] Webserver started at http://127.0.104.3:35275/ using document root <none> and password file <none>
I20250110 02:51:08.102088 975 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:08.102207 975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:08.102531 975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 02:51:08.104853 975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data/instance:
uuid: "8899dd1aeb384e4ea4523f5fd225c888"
format_stamp: "Formatted at 2025-01-10 02:51:08 on dist-test-slave-bpn1"
I20250110 02:51:08.105574 975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/instance:
uuid: "8899dd1aeb384e4ea4523f5fd225c888"
format_stamp: "Formatted at 2025-01-10 02:51:08 on dist-test-slave-bpn1"
I20250110 02:51:08.110391 975 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20250110 02:51:08.113395 993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:08.114532 975 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 02:51:08.114692 975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal
uuid: "8899dd1aeb384e4ea4523f5fd225c888"
format_stamp: "Formatted at 2025-01-10 02:51:08 on dist-test-slave-bpn1"
I20250110 02:51:08.114889 975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:08.125706 465 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Rejecting Update request from peer 5d8d40556dc348e8aa8255d116251a10 for earlier term 2. Current term is 3. Ops: []
I20250110 02:51:08.126361 972 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 02:51:08.126758 971 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 LEADER]: Stepping down as leader of term 2
I20250110 02:51:08.126850 971 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5d8d40556dc348e8aa8255d116251a10, State: Running, Role: LEADER
I20250110 02:51:08.127139 971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:08.127559 971 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 2 FOLLOWER]: Advancing to term 3
I20250110 02:51:08.147644 975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:08.148401 975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:08.148682 975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:08.149596 975 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 02:51:08.152382 975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 02:51:08.152472 975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:08.152593 975 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 02:51:08.152648 975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:08.152856 525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:41089
I20250110 02:51:08.184645 975 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:35771
I20250110 02:51:08.184706 1105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:35771 every 8 connection(s)
I20250110 02:51:08.185824 975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/data/info.pb
I20250110 02:51:08.187942 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 975
I20250110 02:51:08.188120 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/instance
I20250110 02:51:08.212008 465 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Refusing update from remote peer 8bc934cc708045e9858ee8f8e64660cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250110 02:51:08.212926 977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 02:51:08.225415 1116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc934cc708045e9858ee8f8e64660cd. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.225589 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:08.225734 1116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:08.225915 1108 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:08.227144 1108 heartbeater.cc:510] Master 127.0.104.61:44449 requested a full tablet report, sending...
I20250110 02:51:08.228022 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.228271 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:08.228677 525 ts_manager.cc:194] Registered new tserver with Master: 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.229316 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Refusing update from remote peer 8bc934cc708045e9858ee8f8e64660cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250110 02:51:08.229442 607 pending_rounds.cc:77] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Aborting all ops after (but not including) 4
I20250110 02:51:08.229580 607 pending_rounds.cc:99] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250110 02:51:08.229688 607 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250110 02:51:08.229750 525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:54163
I20250110 02:51:08.230463 976 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 02:51:08.233786 1106 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:08.234089 1106 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:08.234705 1106 heartbeater.cc:510] Master 127.0.104.60:39025 requested a full tablet report, sending...
I20250110 02:51:08.235316 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.235579 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:08.235961 971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc934cc708045e9858ee8f8e64660cd. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.236169 971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:08.234812 1107 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35975
I20250110 02:51:08.236653 1107 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:08.236995 1107 heartbeater.cc:510] Master 127.0.104.62:35975 requested a full tablet report, sending...
I20250110 02:51:08.237107 1116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.237547 1116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:08.237829 597 ts_manager.cc:194] Registered new tserver with Master: 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.237998 454 ts_manager.cc:194] Registered new tserver with Master: 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.251555 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 02:51:08.285987 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_779b228790354cc29ced67d72a3f19b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.287782 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_10c8b2eca9024ba393726c84c2782d1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.292759 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_35ffcbc725fc49f0b84935fa73df1a75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.290205 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a3a06170c20844198904336f01011e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.295318 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10c8b2eca9024ba393726c84c2782d1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:08.295341 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3a06170c20844198904336f01011e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.287143 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_44cecb89819340f19bf73e29dc70e228"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.295866 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44cecb89819340f19bf73e29dc70e228 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.288529 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e87e9421f2a34eb1a5d47bf61063105f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.296139 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35ffcbc725fc49f0b84935fa73df1a75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:08.296345 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e87e9421f2a34eb1a5d47bf61063105f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:08.296674 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_779b228790354cc29ced67d72a3f19b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.291558 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e530b09051824d26b72f5a8f5e58f184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.285984 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_fdb2af8f38fa43dea8a9dfca4d02ef1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.298269 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdb2af8f38fa43dea8a9dfca4d02ef1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:08.298097 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e530b09051824d26b72f5a8f5e58f184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.289479 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5991ae19db2f4684a968b2e441da51df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.300055 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5991ae19db2f4684a968b2e441da51df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.289117 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_37363f8917a84999821100ab9e4cda41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.301928 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37363f8917a84999821100ab9e4cda41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.321521 976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:08.321847 976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:08.355778 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5991ae19db2f4684a968b2e441da51df" } new_table_name: "default.Table_3786788ec1f94ce7b71c09c11f0d5ebe" modify_external_catalogs: true
I20250110 02:51:08.357347 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_fdb2af8f38fa43dea8a9dfca4d02ef1c" } new_table_name: "default.Table_1f2bd93f6ba1455daeaef5046baefe0b" modify_external_catalogs: true
I20250110 02:51:08.363445 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_10c8b2eca9024ba393726c84c2782d1c" } new_table_name: "default.Table_f03b7967a5e74371a5b32467b43262bc" modify_external_catalogs: true
I20250110 02:51:08.365643 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e530b09051824d26b72f5a8f5e58f184" } new_table_name: "default.Table_3a960be627b447e0bd193430536685e1" modify_external_catalogs: true
I20250110 02:51:08.367978 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_779b228790354cc29ced67d72a3f19b0" } new_table_name: "default.Table_2ca408d476aa49068e9ddf9a597b2e13" modify_external_catalogs: true
I20250110 02:51:08.368155 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a3a06170c20844198904336f01011e60" } modify_external_catalogs: true
I20250110 02:51:08.368345 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a3a06170c20844198904336f01011e60" } modify_external_catalogs: true
I20250110 02:51:08.368723 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_35ffcbc725fc49f0b84935fa73df1a75" } modify_external_catalogs: true
I20250110 02:51:08.368965 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_35ffcbc725fc49f0b84935fa73df1a75" } modify_external_catalogs: true
I20250110 02:51:08.369475 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_37363f8917a84999821100ab9e4cda41" } modify_external_catalogs: true
I20250110 02:51:08.369688 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_37363f8917a84999821100ab9e4cda41" } modify_external_catalogs: true
I20250110 02:51:08.375941 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskadt4RW/build/asan/bin/kudu pid 564
W20250110 02:51:08.377259 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36fa58b399864d678c81e6bd0d43e790 has no consensus state
I20250110 02:51:08.377318 627 generic_service.cc:196] Checking for leaks (request via RPC)
W20250110 02:51:08.377439 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f34d917babd4b4e8834260fa5dacba4 has no consensus state
W20250110 02:51:08.379449 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 has no consensus state
I20250110 02:51:08.380247 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3786788ec1f94ce7b71c09c11f0d5ebe" } modify_external_catalogs: true
I20250110 02:51:08.380409 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3786788ec1f94ce7b71c09c11f0d5ebe" } modify_external_catalogs: true
I20250110 02:51:08.381111 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1f2bd93f6ba1455daeaef5046baefe0b" } modify_external_catalogs: true
I20250110 02:51:08.381254 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1f2bd93f6ba1455daeaef5046baefe0b" } modify_external_catalogs: true
I20250110 02:51:08.390892 666 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250110 02:51:08.391191 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28ba5987d51a433e886909dffdc3ef8e has no consensus state
W20250110 02:51:08.392033 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce7d45ee5df746d99e38a7cfba334c98 has no consensus state
I20250110 02:51:08.392073 524 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 458d8204747a437c8473cf2b5a5acda2
I20250110 02:51:08.392942 526 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d982b24286134c92bf10fd8a09910775
W20250110 02:51:08.404856 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 consensus state has no leader
I20250110 02:51:08.421252 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 (DEFAULT_TABLE table=default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.422467 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34d917babd4b4e8834260fa5dacba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.424914 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (DEFAULT_TABLE table=default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.425002 894 tablet_service.cc:1467] Processing CreateTablet for tablet 36fa58b399864d678c81e6bd0d43e790 (DEFAULT_TABLE table=default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.425420 895 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 (DEFAULT_TABLE table=default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.425890 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d45ee5df746d99e38a7cfba334c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.426602 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa58b399864d678c81e6bd0d43e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.426930 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.420301 898 tablet_service.cc:1467] Processing CreateTablet for tablet e8541e1c6d454877855c6b1b2e6e334f (DEFAULT_TABLE table=default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.427668 897 tablet_service.cc:1467] Processing CreateTablet for tablet 28ba5987d51a433e886909dffdc3ef8e (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.427898 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8541e1c6d454877855c6b1b2e6e334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.428184 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ba5987d51a433e886909dffdc3ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.434857 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:08.436892 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 36fa58b399864d678c81e6bd0d43e790
I20250110 02:51:08.437563 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet e8541e1c6d454877855c6b1b2e6e334f
I20250110 02:51:08.441828 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.442474 890 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.442749 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.444360 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b (DEFAULT_TABLE table=default.Table_44cecb89819340f19bf73e29dc70e228 [id=1a1e7395b41a4c9cae24c80f746dff63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.445353 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c46be1162124b50ae43d8ea57cb5a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.445607 890 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf (DEFAULT_TABLE table=default.Table_e87e9421f2a34eb1a5d47bf61063105f [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.446112 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.450985 755 tablet_service.cc:1467] Processing CreateTablet for tablet 28ba5987d51a433e886909dffdc3ef8e (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.452896 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ba5987d51a433e886909dffdc3ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.454329 754 tablet_service.cc:1467] Processing CreateTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 (DEFAULT_TABLE table=default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.454793 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34d917babd4b4e8834260fa5dacba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.458802 753 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 (DEFAULT_TABLE table=default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.458559 1041 tablet_service.cc:1467] Processing CreateTablet for tablet e8541e1c6d454877855c6b1b2e6e334f (DEFAULT_TABLE table=default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.459228 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d45ee5df746d99e38a7cfba334c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.459797 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8541e1c6d454877855c6b1b2e6e334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.460192 752 tablet_service.cc:1467] Processing CreateTablet for tablet 36fa58b399864d678c81e6bd0d43e790 (DEFAULT_TABLE table=default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.460701 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa58b399864d678c81e6bd0d43e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.460835 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf (DEFAULT_TABLE table=default.Table_e87e9421f2a34eb1a5d47bf61063105f [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.461287 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.461645 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (DEFAULT_TABLE table=default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.462139 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.462528 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b (DEFAULT_TABLE table=default.Table_44cecb89819340f19bf73e29dc70e228 [id=1a1e7395b41a4c9cae24c80f746dff63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.463004 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c46be1162124b50ae43d8ea57cb5a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.463075 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.463752 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.463903 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 28ba5987d51a433e886909dffdc3ef8e (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.464037 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.464329 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ba5987d51a433e886909dffdc3ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:08.464510 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7f34d917babd4b4e8834260fa5dacba4 because the tablet was not found. No further retry: Not found: Tablet not found: 7f34d917babd4b4e8834260fa5dacba4
W20250110 02:51:08.465014 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e8541e1c6d454877855c6b1b2e6e334f because the tablet was not found. No further retry: Not found: Tablet not found: e8541e1c6d454877855c6b1b2e6e334f
W20250110 02:51:08.465230 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 36fa58b399864d678c81e6bd0d43e790 because the tablet was not found. No further retry: Not found: Tablet not found: 36fa58b399864d678c81e6bd0d43e790
I20250110 02:51:08.465734 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.466030 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b (DEFAULT_TABLE table=default.Table_44cecb89819340f19bf73e29dc70e228 [id=1a1e7395b41a4c9cae24c80f746dff63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.466398 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.466671 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c46be1162124b50ae43d8ea57cb5a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.466697 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.467121 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (DEFAULT_TABLE table=default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.467574 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.467576 749 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf (DEFAULT_TABLE table=default.Table_e87e9421f2a34eb1a5d47bf61063105f [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.468744 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.469381 756 tablet_service.cc:1467] Processing CreateTablet for tablet e8541e1c6d454877855c6b1b2e6e334f (DEFAULT_TABLE table=default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.469898 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8541e1c6d454877855c6b1b2e6e334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.470785 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 (DEFAULT_TABLE table=default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.471210 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34d917babd4b4e8834260fa5dacba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.474292 1038 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 (DEFAULT_TABLE table=default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.474818 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d45ee5df746d99e38a7cfba334c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.478174 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 36fa58b399864d678c81e6bd0d43e790 (DEFAULT_TABLE table=default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.478775 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa58b399864d678c81e6bd0d43e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.483352 1181 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.491384 1181 tablet_bootstrap.cc:654] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.492738 1181 log.cc:826] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:08.499500 1181 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.499756 1181 ts_tablet_manager.cc:1397] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
W20250110 02:51:08.509765 512 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e8541e1c6d454877855c6b1b2e6e334f because the tablet was not found. No further retry: Not found: Tablet not found: e8541e1c6d454877855c6b1b2e6e334f
W20250110 02:51:08.510406 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 36fa58b399864d678c81e6bd0d43e790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36fa58b399864d678c81e6bd0d43e790 already in progress: creating tablet
W20250110 02:51:08.510702 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7f34d917babd4b4e8834260fa5dacba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f34d917babd4b4e8834260fa5dacba4 already in progress: creating tablet
I20250110 02:51:08.514938 1181 raft_consensus.cc:357] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.515350 1181 raft_consensus.cc:383] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.515452 1181 raft_consensus.cc:738] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.516168 1181 consensus_queue.cc:260] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.518242 1184 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:08.521481 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7f34d917babd4b4e8834260fa5dacba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f34d917babd4b4e8834260fa5dacba4 already in progress: creating tablet
W20250110 02:51:08.522065 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 36fa58b399864d678c81e6bd0d43e790 because the tablet was not found. No further retry: Not found: Tablet not found: 36fa58b399864d678c81e6bd0d43e790
W20250110 02:51:08.522295 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e8541e1c6d454877855c6b1b2e6e334f because the tablet was not found. No further retry: Not found: Tablet not found: e8541e1c6d454877855c6b1b2e6e334f
I20250110 02:51:08.523984 1184 tablet_bootstrap.cc:654] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.525693 1184 log.cc:826] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:08.527629 1181 ts_tablet_manager.cc:1428] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.028s user 0.012s sys 0.008s
I20250110 02:51:08.527969 1181 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.528514 1108 heartbeater.cc:502] Master 127.0.104.61:44449 was elected leader, sending a full tablet report...
I20250110 02:51:08.529845 1181 tablet_bootstrap.cc:654] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.532582 1184 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.532807 1184 ts_tablet_manager.cc:1397] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20250110 02:51:08.533190 517 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 1 got 0
I20250110 02:51:08.533439 517 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 28ba5987d51a433e886909dffdc3ef8e (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
I20250110 02:51:08.533609 517 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]). Expected version 1 got 0
I20250110 02:51:08.535761 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.535965 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.537544 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:08.537930 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 36fa58b399864d678c81e6bd0d43e790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36fa58b399864d678c81e6bd0d43e790 already in progress: creating tablet
W20250110 02:51:08.538153 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7f34d917babd4b4e8834260fa5dacba4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f34d917babd4b4e8834260fa5dacba4 already in progress: creating tablet
W20250110 02:51:08.538695 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e8541e1c6d454877855c6b1b2e6e334f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8541e1c6d454877855c6b1b2e6e334f already in progress: creating tablet
I20250110 02:51:08.539095 1181 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.539260 1181 ts_tablet_manager.cc:1397] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250110 02:51:08.540127 1181 raft_consensus.cc:357] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.540359 1181 raft_consensus.cc:383] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.540474 1181 raft_consensus.cc:738] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.540753 1181 consensus_queue.cc:260] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.541337 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d5b4c03572aa4c8fae0339d6ea589418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.541836 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b4c03572aa4c8fae0339d6ea589418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.544378 1184 raft_consensus.cc:357] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.544763 1184 raft_consensus.cc:383] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.544858 1184 raft_consensus.cc:738] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.545313 1181 ts_tablet_manager.cc:1428] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:08.546061 1184 consensus_queue.cc:260] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.548161 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.548378 1180 tablet_replica.cc:331] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:08.548794 1180 raft_consensus.cc:2238] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.549116 1180 raft_consensus.cc:2267] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.549901 1181 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.550282 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9b68cd7d6a4a435fa169e731958bcdff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.550788 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b68cd7d6a4a435fa169e731958bcdff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.551973 1180 ts_tablet_manager.cc:1905] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.554724 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_4ce9591383824513af29f1ca107fd09c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.555238 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ce9591383824513af29f1ca107fd09c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.556313 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_4fd336aafe184c11a74f324e2ac47f8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.556782 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fd336aafe184c11a74f324e2ac47f8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.559162 1180 ts_tablet_manager.cc:1918] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.561162 1180 log.cc:1198] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:08.561605 1180 ts_tablet_manager.cc:1939] T 7f34d917babd4b4e8834260fa5dacba4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:08.563176 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7f34d917babd4b4e8834260fa5dacba4 (table default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]) successfully deleted
I20250110 02:51:08.561028 1181 tablet_bootstrap.cc:654] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.568965 1181 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.569172 1181 ts_tablet_manager.cc:1397] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250110 02:51:08.569164 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0ab1bd2994d6483c9f1a0de6aeca3be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.569684 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ab1bd2994d6483c9f1a0de6aeca3be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.570238 1181 raft_consensus.cc:357] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.570533 1181 raft_consensus.cc:383] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.570660 1181 raft_consensus.cc:738] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.570758 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7d0a3876ce3d48d8b614244462399656"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.571120 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d0a3876ce3d48d8b614244462399656 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.571030 1181 consensus_queue.cc:260] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.571605 755 tablet_service.cc:1467] Processing CreateTablet for tablet 308f307f4200409aaa47df55f20472d0 (DEFAULT_TABLE table=default.Table_9b68cd7d6a4a435fa169e731958bcdff [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.572006 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f307f4200409aaa47df55f20472d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.572746 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 308f307f4200409aaa47df55f20472d0 (DEFAULT_TABLE table=default.Table_9b68cd7d6a4a435fa169e731958bcdff [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.573138 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f307f4200409aaa47df55f20472d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.574352 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]). Expected version 1 got 0
I20250110 02:51:08.574647 748 tablet_service.cc:1467] Processing CreateTablet for tablet 49042c982df448e2b61ce38de50bf1a1 (DEFAULT_TABLE table=default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.574975 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49042c982df448e2b61ce38de50bf1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.575547 1181 ts_tablet_manager.cc:1428] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:08.578178 898 tablet_service.cc:1467] Processing CreateTablet for tablet 49042c982df448e2b61ce38de50bf1a1 (DEFAULT_TABLE table=default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.578606 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49042c982df448e2b61ce38de50bf1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.578573 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]). Expected version 1 got 0
I20250110 02:51:08.579654 1181 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.581284 1181 tablet_bootstrap.cc:654] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.581722 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 49042c982df448e2b61ce38de50bf1a1 (DEFAULT_TABLE table=default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.582044 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49042c982df448e2b61ce38de50bf1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.583436 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e87e9421f2a34eb1a5d47bf61063105f" } new_table_name: "default.Table_b5cee20c2ba547278df5e4649d16f0b0" modify_external_catalogs: true
I20250110 02:51:08.583513 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2e65de9dd8c84b94aa607fce39e93fb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.583925 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e65de9dd8c84b94aa607fce39e93fb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.586783 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e44620c0b9ce431ba206f9111c076233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.587206 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e44620c0b9ce431ba206f9111c076233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.595888 1184 ts_tablet_manager.cc:1428] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.063s user 0.008s sys 0.009s
I20250110 02:51:08.598538 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_44cecb89819340f19bf73e29dc70e228" } new_table_name: "default.Table_f97a80832b8348cc9f5839ebe6887044" modify_external_catalogs: true
I20250110 02:51:08.599185 824 heartbeater.cc:502] Master 127.0.104.61:44449 was elected leader, sending a full tablet report...
I20250110 02:51:08.600216 1184 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.600558 896 tablet_service.cc:1467] Processing CreateTablet for tablet 308f307f4200409aaa47df55f20472d0 (DEFAULT_TABLE table=default.Table_9b68cd7d6a4a435fa169e731958bcdff [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.600886 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f307f4200409aaa47df55f20472d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.601915 1184 tablet_bootstrap.cc:654] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.605675 1188 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.607379 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4ce9591383824513af29f1ca107fd09c" } new_table_name: "default.Table_d24028774b004dbfabe940b74a12bd16" modify_external_catalogs: true
I20250110 02:51:08.608258 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7192835167d842589c56b4b5432a247f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.610491 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7192835167d842589c56b4b5432a247f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.616397 1184 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.616572 1184 ts_tablet_manager.cc:1397] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250110 02:51:08.617784 1184 raft_consensus.cc:357] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.618068 1184 raft_consensus.cc:383] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.618152 1184 raft_consensus.cc:738] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.618469 1184 consensus_queue.cc:260] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.622622 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f03b7967a5e74371a5b32467b43262bc" } new_table_name: "default.Table_accf846f8cb6482db39436051e3fdd6f" modify_external_catalogs: true
I20250110 02:51:08.623945 755 tablet_service.cc:1467] Processing CreateTablet for tablet c81e5a619cd643dea7b457f4916ecf68 (DEFAULT_TABLE table=default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.624358 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81e5a619cd643dea7b457f4916ecf68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.624399 1041 tablet_service.cc:1467] Processing CreateTablet for tablet c81e5a619cd643dea7b457f4916ecf68 (DEFAULT_TABLE table=default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.624444 1040 tablet_service.cc:1467] Processing CreateTablet for tablet c683bc3feaff44b1830cccfe839f677e (DEFAULT_TABLE table=default.Table_4ce9591383824513af29f1ca107fd09c [id=82b5379bd1dc4c09a2aac28e5583e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.624769 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81e5a619cd643dea7b457f4916ecf68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.624940 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c683bc3feaff44b1830cccfe839f677e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.626730 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 1 got 0
I20250110 02:51:08.626928 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]). Expected version 1 got 0
I20250110 02:51:08.627249 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 28ba5987d51a433e886909dffdc3ef8e (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
I20250110 02:51:08.628345 890 tablet_service.cc:1467] Processing CreateTablet for tablet c683bc3feaff44b1830cccfe839f677e (DEFAULT_TABLE table=default.Table_4ce9591383824513af29f1ca107fd09c [id=82b5379bd1dc4c09a2aac28e5583e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.628721 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c683bc3feaff44b1830cccfe839f677e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.630695 748 tablet_service.cc:1467] Processing CreateTablet for tablet c683bc3feaff44b1830cccfe839f677e (DEFAULT_TABLE table=default.Table_4ce9591383824513af29f1ca107fd09c [id=82b5379bd1dc4c09a2aac28e5583e04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.631175 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c683bc3feaff44b1830cccfe839f677e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.632877 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9b68cd7d6a4a435fa169e731958bcdff" } new_table_name: "default.Table_237c5593f10c43598915982e8e3961f7" modify_external_catalogs: true
I20250110 02:51:08.635104 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskadt4RW/build/asan/bin/kudu with pid 564
W20250110 02:51:08.638561 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54fd7ceb38cf413a8b8c6c2e710d4adf consensus state has no leader
I20250110 02:51:08.638892 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d5b4c03572aa4c8fae0339d6ea589418" } modify_external_catalogs: true
I20250110 02:51:08.639034 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d5b4c03572aa4c8fae0339d6ea589418" } modify_external_catalogs: true
I20250110 02:51:08.640405 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b586408fcd3741a3a2b249e88f54eacf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.640831 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b586408fcd3741a3a2b249e88f54eacf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.643633 897 tablet_service.cc:1467] Processing CreateTablet for tablet c81e5a619cd643dea7b457f4916ecf68 (DEFAULT_TABLE table=default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.644034 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81e5a619cd643dea7b457f4916ecf68. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:08.645887 825 tablet.cc:2367] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 02:51:08.655481 965 heartbeater.cc:502] Master 127.0.104.61:44449 was elected leader, sending a full tablet report...
I20250110 02:51:08.663960 1184 ts_tablet_manager.cc:1428] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.047s user 0.003s sys 0.000s
I20250110 02:51:08.664229 1184 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.665800 1184 tablet_bootstrap.cc:654] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.672047 1188 tablet_bootstrap.cc:654] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:08.674273 822 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:39025 (error 108)
W20250110 02:51:08.674305 1106 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:39025 (error 108)
W20250110 02:51:08.680310 1192 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:08.674752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:39025 (local address 127.0.104.1:52973)
0110 02:51:08.678268 (+ 3516us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:08.678287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0110 02:51:08.678425 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 02:51:08.679392 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3409,"thread_start_us":3282,"threads_started":1}
I20250110 02:51:08.680465 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3a960be627b447e0bd193430536685e1" } modify_external_catalogs: true
I20250110 02:51:08.680686 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3a960be627b447e0bd193430536685e1" } modify_external_catalogs: true
W20250110 02:51:08.680891 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 308f307f4200409aaa47df55f20472d0 consensus state has no leader
I20250110 02:51:08.681236 1188 log.cc:826] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:08.681262 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 7f34d917babd4b4e8834260fa5dacba4 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.681969 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 54fd7ceb38cf413a8b8c6c2e710d4adf from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:08.682395 851 connection.cc:537] client connection to 127.0.104.60:39025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 02:51:08.682956 963 heartbeater.cc:643] Failed to heartbeat to 127.0.104.60:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 02:51:08.683353 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 7f34d917babd4b4e8834260fa5dacba4 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.683818 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 28ba5987d51a433e886909dffdc3ef8e from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.685662 1040 tablet_service.cc:1467] Processing CreateTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 (DEFAULT_TABLE table=default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.686210 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe17b2fbe80d4468adf9adb20fd86676. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:08.689462 510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd -> Peer 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025): Couldn't send request to peer 5d8d40556dc348e8aa8255d116251a10. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:08.689989 896 tablet_service.cc:1467] Processing CreateTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 (DEFAULT_TABLE table=default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:08.690757 1109 tablet.cc:2367] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 02:51:08.691040 748 tablet_service.cc:1467] Processing CreateTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 (DEFAULT_TABLE table=default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.691541 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe17b2fbe80d4468adf9adb20fd86676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.692101 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe17b2fbe80d4468adf9adb20fd86676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.694537 1181 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.694738 1181 ts_tablet_manager.cc:1397] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.115s user 0.036s sys 0.049s
I20250110 02:51:08.695645 1181 raft_consensus.cc:357] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.695905 1181 raft_consensus.cc:383] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.696053 1181 raft_consensus.cc:738] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.696429 1181 consensus_queue.cc:260] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.700124 1181 ts_tablet_manager.cc:1428] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:08.700474 1181 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.702708 1181 tablet_bootstrap.cc:654] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:08.705703 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c683bc3feaff44b1830cccfe839f677e consensus state has no leader
I20250110 02:51:08.706701 1184 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.706856 1184 ts_tablet_manager.cc:1397] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.014s
I20250110 02:51:08.707636 1184 raft_consensus.cc:357] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.707841 1184 raft_consensus.cc:383] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.707921 1184 raft_consensus.cc:738] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.708809 1184 consensus_queue.cc:260] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.710009 1184 ts_tablet_manager.cc:1428] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:08.710291 1184 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.712010 1184 tablet_bootstrap.cc:654] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.712951 1188 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.713284 1188 ts_tablet_manager.cc:1397] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.009s
I20250110 02:51:08.717921 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 28ba5987d51a433e886909dffdc3ef8e from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.720335 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 54fd7ceb38cf413a8b8c6c2e710d4adf
I20250110 02:51:08.720898 524 catalog_manager.cc:6596] ReplaceTablet: tablet 54fd7ceb38cf413a8b8c6c2e710d4adf deleted and replaced by tablet e91197c808ba4710970fcdb57f06d4b9
I20250110 02:51:08.721298 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4fd336aafe184c11a74f324e2ac47f8d" } modify_external_catalogs: true
I20250110 02:51:08.721513 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4fd336aafe184c11a74f324e2ac47f8d" } modify_external_catalogs: true
I20250110 02:51:08.722687 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 28ba5987d51a433e886909dffdc3ef8e
I20250110 02:51:08.723203 519 catalog_manager.cc:6596] ReplaceTablet: tablet 28ba5987d51a433e886909dffdc3ef8e deleted and replaced by tablet 80900124f93d43a0abf93175b16225a7
I20250110 02:51:08.723124 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0ab1bd2994d6483c9f1a0de6aeca3be3" } modify_external_catalogs: true
I20250110 02:51:08.723348 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0ab1bd2994d6483c9f1a0de6aeca3be3" } modify_external_catalogs: true
I20250110 02:51:08.723534 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7d0a3876ce3d48d8b614244462399656" } modify_external_catalogs: true
I20250110 02:51:08.723716 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7d0a3876ce3d48d8b614244462399656" } modify_external_catalogs: true
I20250110 02:51:08.725291 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 49042c982df448e2b61ce38de50bf1a1
I20250110 02:51:08.728004 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ba5987d51a433e886909dffdc3ef8e with delete_type TABLET_DATA_DELETED (replaced by tablet 80900124f93d43a0abf93175b16225a7) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.728283 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.728449 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf with delete_type TABLET_DATA_DELETED (replaced by tablet e91197c808ba4710970fcdb57f06d4b9) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:08.729460 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 28ba5987d51a433e886909dffdc3ef8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28ba5987d51a433e886909dffdc3ef8e already in progress: creating tablet
W20250110 02:51:08.729714 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 49042c982df448e2b61ce38de50bf1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49042c982df448e2b61ce38de50bf1a1 already in progress: creating tablet
I20250110 02:51:08.729746 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf with delete_type TABLET_DATA_DELETED (replaced by tablet e91197c808ba4710970fcdb57f06d4b9) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:08.729947 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54fd7ceb38cf413a8b8c6c2e710d4adf already in progress: creating tablet
I20250110 02:51:08.729986 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ba5987d51a433e886909dffdc3ef8e with delete_type TABLET_DATA_DELETED (replaced by tablet 80900124f93d43a0abf93175b16225a7) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.730175 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.730419 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf with delete_type TABLET_DATA_DELETED (replaced by tablet e91197c808ba4710970fcdb57f06d4b9) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.730710 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ba5987d51a433e886909dffdc3ef8e with delete_type TABLET_DATA_DELETED (replaced by tablet 80900124f93d43a0abf93175b16225a7) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.730919 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.731359 1179 tablet_replica.cc:331] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:08.731812 1179 raft_consensus.cc:2238] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.732193 1179 raft_consensus.cc:2267] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.732610 1180 tablet_replica.cc:331] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:08.732790 1180 raft_consensus.cc:2238] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.732985 1180 raft_consensus.cc:2267] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.734261 1180 ts_tablet_manager.cc:1905] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.734817 1188 raft_consensus.cc:357] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.735335 1188 raft_consensus.cc:383] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.735450 1188 raft_consensus.cc:738] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:08.736055 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54fd7ceb38cf413a8b8c6c2e710d4adf already in progress: creating tablet
I20250110 02:51:08.737479 517 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8e8b34718e754371a286e9cbdc85c2af
I20250110 02:51:08.738680 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:08.739333 896 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.739776 1179 ts_tablet_manager.cc:1905] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:08.739946 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fe17b2fbe80d4468adf9adb20fd86676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe17b2fbe80d4468adf9adb20fd86676 already in progress: creating tablet
I20250110 02:51:08.740530 756 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.741078 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.741063 1188 consensus_queue.cc:260] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.742291 1181 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.742467 1181 ts_tablet_manager.cc:1397] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.020s
I20250110 02:51:08.742877 1188 ts_tablet_manager.cc:1428] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.030s user 0.016s sys 0.001s
I20250110 02:51:08.743196 1181 raft_consensus.cc:357] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.743393 1181 raft_consensus.cc:383] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.743463 1181 raft_consensus.cc:738] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.743705 1181 consensus_queue.cc:260] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.744194 1181 ts_tablet_manager.cc:1428] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:08.744542 1181 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.746053 1180 ts_tablet_manager.cc:1918] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.746156 1181 tablet_bootstrap.cc:654] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.746209 1180 log.cc:1198] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/54fd7ceb38cf413a8b8c6c2e710d4adf
I20250110 02:51:08.746606 1180 ts_tablet_manager.cc:1939] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:08.748898 1179 ts_tablet_manager.cc:1918] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.749095 1179 log.cc:1198] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/28ba5987d51a433e886909dffdc3ef8e
I20250110 02:51:08.749554 1179 ts_tablet_manager.cc:1939] T 28ba5987d51a433e886909dffdc3ef8e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:08.750427 1184 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.752049 1184 ts_tablet_manager.cc:1397] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.004s
I20250110 02:51:08.752815 1184 raft_consensus.cc:357] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.753053 1184 raft_consensus.cc:383] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.753535 1184 raft_consensus.cc:738] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.753919 1184 consensus_queue.cc:260] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.755160 1184 ts_tablet_manager.cc:1428] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250110 02:51:08.755450 1184 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.756366 1188 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.757236 1184 tablet_bootstrap.cc:654] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.758569 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_fb098e79694d4feaafc62a7c3d9eef8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.759124 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb098e79694d4feaafc62a7c3d9eef8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.759173 1181 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.759287 1181 ts_tablet_manager.cc:1397] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20250110 02:51:08.760030 1181 raft_consensus.cc:357] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.760244 1181 raft_consensus.cc:383] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.760353 1181 raft_consensus.cc:738] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.760596 1181 consensus_queue.cc:260] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.760658 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 28ba5987d51a433e886909dffdc3ef8e (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
W20250110 02:51:08.760972 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 49042c982df448e2b61ce38de50bf1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49042c982df448e2b61ce38de50bf1a1 already in progress: creating tablet
W20250110 02:51:08.761198 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fe17b2fbe80d4468adf9adb20fd86676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe17b2fbe80d4468adf9adb20fd86676 already in progress: creating tablet
I20250110 02:51:08.762467 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3edf95be0d264f409133f83c3e948d58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.762705 756 tablet_service.cc:1467] Processing CreateTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd (DEFAULT_TABLE table=default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:08.762934 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3edf95be0d264f409133f83c3e948d58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.762971 1181 ts_tablet_manager.cc:1428] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:08.763105 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b27ee8bd87f4316baccc8b85c84d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.763900 1181 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.765479 1181 tablet_bootstrap.cc:654] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.766809 755 tablet_service.cc:1467] Processing CreateTablet for tablet 24efc81a5ac743809665b17f941c0652 (DEFAULT_TABLE table=default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.767328 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24efc81a5ac743809665b17f941c0652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.767413 1188 tablet_bootstrap.cc:654] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.768688 1181 tablet_bootstrap.cc:492] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.768873 1181 ts_tablet_manager.cc:1397] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250110 02:51:08.769721 1181 raft_consensus.cc:357] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.769963 1181 raft_consensus.cc:383] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.770047 1181 raft_consensus.cc:738] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.770284 1181 consensus_queue.cc:260] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.770965 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 24efc81a5ac743809665b17f941c0652 (DEFAULT_TABLE table=default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.771400 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24efc81a5ac743809665b17f941c0652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.772224 1181 ts_tablet_manager.cc:1428] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:08.773295 1181 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.773523 896 tablet_service.cc:1467] Processing CreateTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd (DEFAULT_TABLE table=default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.773878 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b27ee8bd87f4316baccc8b85c84d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.774451 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.774817 1181 tablet_bootstrap.cc:654] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.775753 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd (DEFAULT_TABLE table=default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.776275 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b27ee8bd87f4316baccc8b85c84d2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.778918 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_66eca48354cf4d099ca9f06309db7fe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.779417 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66eca48354cf4d099ca9f06309db7fe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.781129 896 tablet_service.cc:1467] Processing CreateTablet for tablet 24efc81a5ac743809665b17f941c0652 (DEFAULT_TABLE table=default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.781437 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24efc81a5ac743809665b17f941c0652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.789268 1188 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.789454 1188 ts_tablet_manager.cc:1397] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250110 02:51:08.789568 1184 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.789762 1184 ts_tablet_manager.cc:1397] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.027s
I20250110 02:51:08.790488 1188 raft_consensus.cc:357] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.790740 1188 raft_consensus.cc:383] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.790634 1184 raft_consensus.cc:357] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.790884 1184 raft_consensus.cc:383] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.790890 1188 raft_consensus.cc:738] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.790977 1184 raft_consensus.cc:738] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.791195 1188 consensus_queue.cc:260] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.791273 1184 consensus_queue.cc:260] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.791967 1184 ts_tablet_manager.cc:1428] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:08.793380 1184 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.794811 1188 ts_tablet_manager.cc:1428] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:08.795787 1188 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.797343 1188 tablet_bootstrap.cc:654] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.799150 1188 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.800719 1184 tablet_bootstrap.cc:654] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.801285 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f9e91d9b595e465280715b3ca3b05a09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.801806 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9e91d9b595e465280715b3ca3b05a09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.802307 1180 tablet_replica.cc:331] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:08.802521 1180 raft_consensus.cc:2238] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.802711 1180 raft_consensus.cc:2267] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.803664 1180 ts_tablet_manager.cc:1905] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.803859 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 54fd7ceb38cf413a8b8c6c2e710d4adf (table default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]) successfully deleted
I20250110 02:51:08.805631 1188 ts_tablet_manager.cc:1397] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250110 02:51:08.806504 1188 raft_consensus.cc:357] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.806780 1188 raft_consensus.cc:383] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.806923 1188 raft_consensus.cc:738] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.807425 1188 consensus_queue.cc:260] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.807950 1188 ts_tablet_manager.cc:1428] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:08.809932 1188 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.811066 1180 ts_tablet_manager.cc:1918] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.811205 1180 log.cc:1198] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/28ba5987d51a433e886909dffdc3ef8e
I20250110 02:51:08.811652 1180 ts_tablet_manager.cc:1939] T 28ba5987d51a433e886909dffdc3ef8e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:08.812348 1188 tablet_bootstrap.cc:654] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.812826 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 28ba5987d51a433e886909dffdc3ef8e (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
W20250110 02:51:08.813835 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 49042c982df448e2b61ce38de50bf1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49042c982df448e2b61ce38de50bf1a1 already in progress: creating tablet
I20250110 02:51:08.813899 1180 tablet_replica.cc:331] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:08.814071 1180 raft_consensus.cc:2238] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:08.814075 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fe17b2fbe80d4468adf9adb20fd86676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe17b2fbe80d4468adf9adb20fd86676 already in progress: creating tablet
I20250110 02:51:08.814229 1180 raft_consensus.cc:2267] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.814806 1184 tablet_bootstrap.cc:492] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.814893 1180 ts_tablet_manager.cc:1905] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.814951 1184 ts_tablet_manager.cc:1397] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.005s
I20250110 02:51:08.815671 1184 raft_consensus.cc:357] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.815866 1184 raft_consensus.cc:383] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.815953 1184 raft_consensus.cc:738] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.816200 1184 consensus_queue.cc:260] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.816736 1184 ts_tablet_manager.cc:1428] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:08.817034 1184 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.818693 1184 tablet_bootstrap.cc:654] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.820992 1180 ts_tablet_manager.cc:1918] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.821166 1180 log.cc:1198] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/36fa58b399864d678c81e6bd0d43e790
I20250110 02:51:08.821606 1180 ts_tablet_manager.cc:1939] T 36fa58b399864d678c81e6bd0d43e790 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:08.822985 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 36fa58b399864d678c81e6bd0d43e790 (table default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]) successfully deleted
I20250110 02:51:08.823422 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 (DEFAULT_TABLE table=default.Table_b586408fcd3741a3a2b249e88f54eacf [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.823733 1032 tablet_service.cc:1467] Processing CreateTablet for tablet e91197c808ba4710970fcdb57f06d4b9 (DEFAULT_TABLE table=default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.823808 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7422cfed02fc4b50ab867fdf34769b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.824052 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91197c808ba4710970fcdb57f06d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.824106 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 80900124f93d43a0abf93175b16225a7 (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.824386 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80900124f93d43a0abf93175b16225a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.824326 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 (DEFAULT_TABLE table=default.Table_b586408fcd3741a3a2b249e88f54eacf [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.824647 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7422cfed02fc4b50ab867fdf34769b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.826697 898 tablet_service.cc:1467] Processing CreateTablet for tablet 80900124f93d43a0abf93175b16225a7 (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.827091 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80900124f93d43a0abf93175b16225a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.828187 1033 tablet_service.cc:1467] Processing CreateTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (DEFAULT_TABLE table=default.Table_e44620c0b9ce431ba206f9111c076233 [id=b94c35cd0cd3480cb751f1d841850f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.828552 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.830277 749 tablet_service.cc:1467] Processing CreateTablet for tablet e91197c808ba4710970fcdb57f06d4b9 (DEFAULT_TABLE table=default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.830413 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 (DEFAULT_TABLE table=default.Table_b586408fcd3741a3a2b249e88f54eacf [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.830618 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91197c808ba4710970fcdb57f06d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.830735 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7422cfed02fc4b50ab867fdf34769b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.834650 890 tablet_service.cc:1467] Processing CreateTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (DEFAULT_TABLE table=default.Table_e44620c0b9ce431ba206f9111c076233 [id=b94c35cd0cd3480cb751f1d841850f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.834868 756 tablet_service.cc:1467] Processing CreateTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (DEFAULT_TABLE table=default.Table_e44620c0b9ce431ba206f9111c076233 [id=b94c35cd0cd3480cb751f1d841850f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.834972 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.835165 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.838693 748 tablet_service.cc:1467] Processing CreateTablet for tablet 80900124f93d43a0abf93175b16225a7 (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.838745 897 tablet_service.cc:1467] Processing CreateTablet for tablet e91197c808ba4710970fcdb57f06d4b9 (DEFAULT_TABLE table=default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.839013 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80900124f93d43a0abf93175b16225a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.838881 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_50073d844fb342e19b52d0914b56098f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.839061 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91197c808ba4710970fcdb57f06d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:08.839399 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50073d844fb342e19b52d0914b56098f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.841051 1187 raft_consensus.cc:491] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:08.841245 1187 raft_consensus.cc:513] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.841734 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7670a09cdcf64f9c9c258bcda9f0d69f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.842164 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7670a09cdcf64f9c9c258bcda9f0d69f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.842999 1188 tablet_bootstrap.cc:492] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.843209 1187 leader_election.cc:290] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.843245 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_237c5593f10c43598915982e8e3961f7" } new_table_name: "default.Table_cce012e47fbc46f4807cb0cd9adc5568" modify_external_catalogs: true
I20250110 02:51:08.843990 1188 ts_tablet_manager.cc:1397] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.003s
I20250110 02:51:08.849166 1188 raft_consensus.cc:357] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.850062 1181 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.850198 1181 ts_tablet_manager.cc:1397] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.010s
I20250110 02:51:08.850194 1188 raft_consensus.cc:383] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.850385 1188 raft_consensus.cc:738] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.850674 1188 consensus_queue.cc:260] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.850893 1181 raft_consensus.cc:357] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.851075 1181 raft_consensus.cc:383] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.851146 1181 raft_consensus.cc:738] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.851338 1181 consensus_queue.cc:260] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.851739 1181 ts_tablet_manager.cc:1428] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 02:51:08.852005 1181 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.853595 1181 tablet_bootstrap.cc:654] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.857465 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f34d917babd4b4e8834260fa5dacba4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:08.857975 918 raft_consensus.cc:2463] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:08.858863 709 leader_election.cc:304] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:08.859263 1187 raft_consensus.cc:2798] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:08.859414 1188 ts_tablet_manager.cc:1428] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250110 02:51:08.859405 1187 raft_consensus.cc:491] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:08.859504 1187 raft_consensus.cc:3054] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.861287 1188 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.861925 1187 raft_consensus.cc:513] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.863039 1188 tablet_bootstrap.cc:654] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.865290 1184 tablet_bootstrap.cc:492] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.865499 1184 ts_tablet_manager.cc:1397] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20250110 02:51:08.866536 1184 raft_consensus.cc:357] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.866823 1184 raft_consensus.cc:383] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.866918 1184 raft_consensus.cc:738] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.867187 1184 consensus_queue.cc:260] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:08.868130 1188 tablet_bootstrap.cc:492] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.868299 1188 ts_tablet_manager.cc:1397] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250110 02:51:08.870303 1188 raft_consensus.cc:357] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.870633 1188 raft_consensus.cc:383] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.870762 1188 raft_consensus.cc:738] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.871091 1188 consensus_queue.cc:260] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.874051 1188 ts_tablet_manager.cc:1428] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:08.876130 1188 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.877753 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f34d917babd4b4e8834260fa5dacba4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:08.877957 918 raft_consensus.cc:3054] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.878346 1188 tablet_bootstrap.cc:654] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.880942 918 raft_consensus.cc:2463] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:08.881379 709 leader_election.cc:304] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:08.883076 1214 raft_consensus.cc:2798] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:08.883334 1214 raft_consensus.cc:695] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
W20250110 02:51:08.884286 966 tablet.cc:2367] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 02:51:08.885457 1184 ts_tablet_manager.cc:1428] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.020s user 0.008s sys 0.002s
W20250110 02:51:08.885843 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 308f307f4200409aaa47df55f20472d0 consensus state has no leader
I20250110 02:51:08.886047 1184 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.889091 755 tablet_service.cc:1467] Processing CreateTablet for tablet b85eea2ad540445dbd95a655135dc043 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.889468 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85eea2ad540445dbd95a655135dc043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.890790 890 tablet_service.cc:1467] Processing CreateTablet for tablet b85eea2ad540445dbd95a655135dc043 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.891151 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85eea2ad540445dbd95a655135dc043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.892724 1184 tablet_bootstrap.cc:654] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.892711 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ff27131e55134787bfd88b27e7a86c63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.893204 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff27131e55134787bfd88b27e7a86c63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.894357 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d4d87a97dad24a4784c3a82e4bf288f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.894845 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4d87a97dad24a4784c3a82e4bf288f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.895234 896 tablet_service.cc:1467] Processing CreateTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.895606 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b851ddf27fe44d19b9ca39d82e6352cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.896239 748 tablet_service.cc:1467] Processing CreateTablet for tablet 286aee6074e14c27b17bc79a78906620 (DEFAULT_TABLE table=default.Table_66eca48354cf4d099ca9f06309db7fe1 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.896566 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 286aee6074e14c27b17bc79a78906620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.896560 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_66eca48354cf4d099ca9f06309db7fe1" } new_table_name: "default.Table_5381a3601aa745659f017c21643544c5" modify_external_catalogs: true
I20250110 02:51:08.896749 1040 tablet_service.cc:1467] Processing CreateTablet for tablet b85eea2ad540445dbd95a655135dc043 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.897089 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85eea2ad540445dbd95a655135dc043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.882328 1187 leader_election.cc:290] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.901119 1198 raft_consensus.cc:491] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:08.901261 1198 raft_consensus.cc:513] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.902693 1032 tablet_service.cc:1467] Processing CreateTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.902983 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b851ddf27fe44d19b9ca39d82e6352cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.903165 1198 leader_election.cc:290] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.903844 755 tablet_service.cc:1467] Processing CreateTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.904193 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b851ddf27fe44d19b9ca39d82e6352cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.905833 897 tablet_service.cc:1467] Processing CreateTablet for tablet 286aee6074e14c27b17bc79a78906620 (DEFAULT_TABLE table=default.Table_66eca48354cf4d099ca9f06309db7fe1 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.906646 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 286aee6074e14c27b17bc79a78906620 (DEFAULT_TABLE table=default.Table_66eca48354cf4d099ca9f06309db7fe1 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.906960 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 286aee6074e14c27b17bc79a78906620. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.910744 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 286aee6074e14c27b17bc79a78906620. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:08.912567 711 leader_election.cc:343] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 7f34d917babd4b4e8834260fa5dacba4
W20250110 02:51:08.913197 711 leader_election.cc:343] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:08.915184 1184 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.915323 1184 ts_tablet_manager.cc:1397] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20250110 02:51:08.916121 1184 raft_consensus.cc:357] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.916385 1184 raft_consensus.cc:383] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.916504 1184 raft_consensus.cc:738] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.916747 1184 consensus_queue.cc:260] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.917459 1184 ts_tablet_manager.cc:1428] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:08.917726 1184 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.919065 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b164fa0aebbe4bb2821b21038f738d0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:08.919605 1184 tablet_bootstrap.cc:654] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:08.919656 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b164fa0aebbe4bb2821b21038f738d0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.920746 748 tablet_service.cc:1467] Processing CreateTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 (DEFAULT_TABLE table=default.Table_f9e91d9b595e465280715b3ca3b05a09 [id=1da07435b54849318beda7e8518f0bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.921089 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89984fa8f39641a39969f4e3392ae8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.919678 521 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_f03b7967a5e74371a5b32467b43262bc [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 1 got 0
W20250110 02:51:08.925122 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c46be1162124b50ae43d8ea57cb5a0b consensus state has no leader
I20250110 02:51:08.925385 520 master_service.cc:946] ReplaceTablet: received request to replace tablet c81e5a619cd643dea7b457f4916ecf68 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.925442 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 (DEFAULT_TABLE table=default.Table_f9e91d9b595e465280715b3ca3b05a09 [id=1da07435b54849318beda7e8518f0bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.925763 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 54fd7ceb38cf413a8b8c6c2e710d4adf from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.925855 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89984fa8f39641a39969f4e3392ae8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.926205 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f9e91d9b595e465280715b3ca3b05a09" } new_table_name: "default.Table_46bcf45f1fce419c882b8c239e680ee9" modify_external_catalogs: true
W20250110 02:51:08.927457 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 consensus state has no leader
I20250110 02:51:08.927503 1181 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:08.927647 1181 ts_tablet_manager.cc:1397] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.021s
I20250110 02:51:08.927794 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6b27d5e1d8994346acba31f9770550b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:08.928220 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b27d5e1d8994346acba31f9770550b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:08.928490 1181 raft_consensus.cc:357] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.929005 897 tablet_service.cc:1467] Processing CreateTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 (DEFAULT_TABLE table=default.Table_f9e91d9b595e465280715b3ca3b05a09 [id=1da07435b54849318beda7e8518f0bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.929111 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet ce7d45ee5df746d99e38a7cfba334c98
I20250110 02:51:08.928691 1181 raft_consensus.cc:383] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.929405 1181 raft_consensus.cc:738] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.929680 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c81e5a619cd643dea7b457f4916ecf68
I20250110 02:51:08.929695 1181 consensus_queue.cc:260] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.930176 1181 ts_tablet_manager.cc:1428] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:08.930564 1181 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:08.930809 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 (DEFAULT_TABLE table=default.Table_50073d844fb342e19b52d0914b56098f [id=dd4008ea14ad40c68b3e71fac9e2be88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.931157 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10504e77a5c64ebf85be937d64ca9c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.932436 1181 tablet_bootstrap.cc:654] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.939461 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b27ee8bd87f4316baccc8b85c84d2fd from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.939675 1187 raft_consensus.cc:491] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:08.939839 1187 raft_consensus.cc:513] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.940519 1187 leader_election.cc:290] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.940538 755 tablet_service.cc:1467] Processing CreateTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 (DEFAULT_TABLE table=default.Table_50073d844fb342e19b52d0914b56098f [id=dd4008ea14ad40c68b3e71fac9e2be88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.942324 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10504e77a5c64ebf85be937d64ca9c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.942822 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d45ee5df746d99e38a7cfba334c98" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:08.943838 709 leader_election.cc:343] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:08.944279 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89984fa8f39641a39969f4e3392ae8b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.946079 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d45ee5df746d99e38a7cfba334c98" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:08.921557 896 tablet_service.cc:1467] Processing CreateTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 (DEFAULT_TABLE table=default.Table_50073d844fb342e19b52d0914b56098f [id=dd4008ea14ad40c68b3e71fac9e2be88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:08.946682 1061 raft_consensus.cc:2463] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:08.946782 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10504e77a5c64ebf85be937d64ca9c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:08.947271 711 leader_election.cc:304] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:08.947754 1187 raft_consensus.cc:2798] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:08.947896 1187 raft_consensus.cc:491] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:08.948000 1187 raft_consensus.cc:3054] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.948719 517 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]). Expected version 1 got 0
I20250110 02:51:08.950381 1187 raft_consensus.cc:513] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.951191 1187 leader_election.cc:290] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.951776 523 master_service.cc:946] ReplaceTablet: received request to replace tablet 7f34d917babd4b4e8834260fa5dacba4 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.952386 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2e65de9dd8c84b94aa607fce39e93fb9" } modify_external_catalogs: true
I20250110 02:51:08.952621 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2e65de9dd8c84b94aa607fce39e93fb9" } modify_external_catalogs: true
I20250110 02:51:08.953979 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ff27131e55134787bfd88b27e7a86c63" } modify_external_catalogs: true
I20250110 02:51:08.954216 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ff27131e55134787bfd88b27e7a86c63" } modify_external_catalogs: true
I20250110 02:51:08.955520 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d45ee5df746d99e38a7cfba334c98" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:08.955762 1061 raft_consensus.cc:3054] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.957218 1184 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:08.957418 1184 ts_tablet_manager.cc:1397] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.006s
I20250110 02:51:08.958323 1184 raft_consensus.cc:357] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.958649 1184 raft_consensus.cc:383] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.958751 1184 raft_consensus.cc:738] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.959110 1184 consensus_queue.cc:260] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.960654 1184 ts_tablet_manager.cc:1428] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:08.960757 521 master_service.cc:946] ReplaceTablet: received request to replace tablet b851ddf27fe44d19b9ca39d82e6352cd from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:08.961915 1184 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:08.963937 1184 tablet_bootstrap.cc:654] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.969394 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.969594 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.969933 1179 tablet_replica.cc:331] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:08.970127 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.884320 1214 consensus_queue.cc:237] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:08.970520 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c81e5a619cd643dea7b457f4916ecf68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c81e5a619cd643dea7b457f4916ecf68 already in progress: creating tablet
I20250110 02:51:08.970088 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd7ceb38cf413a8b8c6c2e710d4adf" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:08.970731 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.970927 776 raft_consensus.cc:2463] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:08.970543 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.971541 853 leader_election.cc:304] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:08.971622 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.971881 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.971920 1198 raft_consensus.cc:2798] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:08.972044 1198 raft_consensus.cc:491] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:08.972076 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.972134 1198 raft_consensus.cc:3054] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.972560 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.972784 1180 tablet_replica.cc:331] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:08.972956 1180 raft_consensus.cc:2238] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.973127 1180 raft_consensus.cc:2267] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.973430 1179 raft_consensus.cc:2238] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.973608 1179 raft_consensus.cc:2267] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:08.974623 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 286aee6074e14c27b17bc79a78906620 consensus state has no leader
I20250110 02:51:08.974788 1198 raft_consensus.cc:513] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.975315 1179 ts_tablet_manager.cc:1905] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.975665 1198 leader_election.cc:290] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:08.975772 519 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 2 got 0
I20250110 02:51:08.976534 1188 tablet_bootstrap.cc:492] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:08.976740 1188 ts_tablet_manager.cc:1397] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.005s
I20250110 02:51:08.977627 1188 raft_consensus.cc:357] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.977931 1188 raft_consensus.cc:383] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:08.978087 1180 ts_tablet_manager.cc:1905] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.978262 1188 raft_consensus.cc:738] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:08.978613 1188 consensus_queue.cc:260] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:08.979229 1188 ts_tablet_manager.cc:1428] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:08.980185 1188 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:08.982033 1188 tablet_bootstrap.cc:654] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:08.983361 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b5cee20c2ba547278df5e4649d16f0b0" } modify_external_catalogs: true
I20250110 02:51:08.983595 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b5cee20c2ba547278df5e4649d16f0b0" } modify_external_catalogs: true
I20250110 02:51:08.985535 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 3b27ee8bd87f4316baccc8b85c84d2fd
I20250110 02:51:08.985971 1179 ts_tablet_manager.cc:1918] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.986019 524 catalog_manager.cc:6596] ReplaceTablet: tablet 3b27ee8bd87f4316baccc8b85c84d2fd deleted and replaced by tablet 9f94ea0136d24be2bd5b3dac9b7b16b7
I20250110 02:51:08.986115 1179 log.cc:1198] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e8541e1c6d454877855c6b1b2e6e334f
I20250110 02:51:08.986526 1179 ts_tablet_manager.cc:1939] T e8541e1c6d454877855c6b1b2e6e334f P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:08.986608 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89984fa8f39641a39969f4e3392ae8b7 consensus state has no leader
I20250110 02:51:08.987937 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (replaced by tablet 9f94ea0136d24be2bd5b3dac9b7b16b7) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:08.988341 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e8541e1c6d454877855c6b1b2e6e334f (table default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]) successfully deleted
I20250110 02:51:08.988538 1180 ts_tablet_manager.cc:1918] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:08.988690 1180 log.cc:1198] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e8541e1c6d454877855c6b1b2e6e334f
I20250110 02:51:08.989068 1180 ts_tablet_manager.cc:1939] T e8541e1c6d454877855c6b1b2e6e334f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:08.989308 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (replaced by tablet 9f94ea0136d24be2bd5b3dac9b7b16b7) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.990031 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e8541e1c6d454877855c6b1b2e6e334f (table default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]) successfully deleted
I20250110 02:51:08.990327 1179 tablet_replica.cc:331] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:08.990556 1179 raft_consensus.cc:2238] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:08.990625 1061 raft_consensus.cc:2463] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:08.990746 1179 raft_consensus.cc:2267] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:08.991746 1179 ts_tablet_manager.cc:1905] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:08.993971 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d45ee5df746d99e38a7cfba334c98" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:08.994163 918 raft_consensus.cc:3054] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.997349 918 raft_consensus.cc:2463] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:08.997992 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd7ceb38cf413a8b8c6c2e710d4adf" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:08.998212 776 raft_consensus.cc:3054] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:08.998219 897 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.998777 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.999137 518 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 600681ec4eb547d989d5d45e6b1bdd8f
I20250110 02:51:08.999153 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf with delete_type TABLET_DATA_DELETED (replaced by tablet e91197c808ba4710970fcdb57f06d4b9) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:08.999457 897 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.000124 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.000555 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ba5987d51a433e886909dffdc3ef8e with delete_type TABLET_DATA_DELETED (replaced by tablet 80900124f93d43a0abf93175b16225a7) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.000895 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:09.000924 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.001271 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (replaced by tablet 9f94ea0136d24be2bd5b3dac9b7b16b7) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.001698 521 catalog_manager.cc:6596] ReplaceTablet: tablet b851ddf27fe44d19b9ca39d82e6352cd deleted and replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0
I20250110 02:51:09.001777 897 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.001969 1178 tablet_replica.cc:331] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.002256 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:08.991160 1180 tablet_replica.cc:331] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.002583 709 leader_election.cc:304] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.002701 1180 raft_consensus.cc:2238] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.002938 1180 raft_consensus.cc:2267] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.003006 1187 raft_consensus.cc:2798] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.003180 1187 raft_consensus.cc:695] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.003568 1187 consensus_queue.cc:237] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.004418 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.005156 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.005798 1180 ts_tablet_manager.cc:1905] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.008123 1179 ts_tablet_manager.cc:1918] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.008219 1178 raft_consensus.cc:2238] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.008253 1179 log.cc:1198] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/36fa58b399864d678c81e6bd0d43e790
I20250110 02:51:09.008610 1179 ts_tablet_manager.cc:1939] T 36fa58b399864d678c81e6bd0d43e790 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.008584 1178 raft_consensus.cc:2267] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.009014 1179 tablet_replica.cc:331] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.009277 1179 raft_consensus.cc:2238] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:09.009651 1179 pending_rounds.cc:62] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:09.009820 1179 pending_rounds.cc:69] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611712913719296 op_type: NO_OP noop_request { }
I20250110 02:51:09.009955 1179 raft_consensus.cc:2883] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:09.010074 1179 raft_consensus.cc:2267] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.010357 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:09.010607 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:09.011085 1178 ts_tablet_manager.cc:1905] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.013252 1180 ts_tablet_manager.cc:1918] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.013392 1180 log.cc:1198] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/ce7d45ee5df746d99e38a7cfba334c98
I20250110 02:51:09.013427 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1eb291591c244ffa92efc9d213a760b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.013768 1180 ts_tablet_manager.cc:1939] T ce7d45ee5df746d99e38a7cfba334c98 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:09.014056 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eb291591c244ffa92efc9d213a760b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.014871 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]) successfully deleted
W20250110 02:51:09.015553 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c81e5a619cd643dea7b457f4916ecf68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c81e5a619cd643dea7b457f4916ecf68 already in progress: creating tablet
I20250110 02:51:09.015872 1180 tablet_replica.cc:331] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.015725 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ad166793c0b34c8bb6bceca5b8580839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.016038 1180 raft_consensus.cc:2238] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.016211 1180 raft_consensus.cc:2267] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:09.016299 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad166793c0b34c8bb6bceca5b8580839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.016934 1180 ts_tablet_manager.cc:1905] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.017496 1178 ts_tablet_manager.cc:1918] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.017652 1178 log.cc:1198] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/ce7d45ee5df746d99e38a7cfba334c98
I20250110 02:51:09.018091 1178 ts_tablet_manager.cc:1939] T ce7d45ee5df746d99e38a7cfba334c98 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.019770 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]) successfully deleted
I20250110 02:51:09.020334 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 36fa58b399864d678c81e6bd0d43e790 (table default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]) successfully deleted
I20250110 02:51:09.020660 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.021100 1179 ts_tablet_manager.cc:1905] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.021852 776 raft_consensus.cc:2463] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:09.023707 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c81e5a619cd643dea7b457f4916ecf68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c81e5a619cd643dea7b457f4916ecf68 already in progress: creating tablet
I20250110 02:51:09.005872 1181 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.024066 1178 tablet_replica.cc:331] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.024061 1181 ts_tablet_manager.cc:1397] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.024s
I20250110 02:51:09.025017 1181 raft_consensus.cc:357] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.025276 1181 raft_consensus.cc:383] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.025202 1178 raft_consensus.cc:2238] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.025382 1181 raft_consensus.cc:738] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.025696 1178 raft_consensus.cc:2267] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.025633 1181 consensus_queue.cc:260] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.026643 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet e91197c808ba4710970fcdb57f06d4b9
W20250110 02:51:09.026942 853 leader_election.cc:343] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 54fd7ceb38cf413a8b8c6c2e710d4adf
I20250110 02:51:09.028882 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.027632 853 leader_election.cc:304] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:09.029317 853 raft_consensus.cc:2681] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 02:51:09.029589 853 leader_election.cc:343] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 54fd7ceb38cf413a8b8c6c2e710d4adf
I20250110 02:51:09.029704 523 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 9f94ea0136d24be2bd5b3dac9b7b16b7
I20250110 02:51:09.032054 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f4f45d5a3c9748859376185f7b09115d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.032519 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4f45d5a3c9748859376185f7b09115d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.033274 1178 ts_tablet_manager.cc:1905] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.034209 897 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.034778 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.034971 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.037425 1179 ts_tablet_manager.cc:1918] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:09.037557 1179 log.cc:1198] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:09.038003 1179 ts_tablet_manager.cc:1939] T 7f34d917babd4b4e8834260fa5dacba4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.038616 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7f34d917babd4b4e8834260fa5dacba4 (table default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]) successfully deleted
I20250110 02:51:09.039789 1181 ts_tablet_manager.cc:1428] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250110 02:51:09.040330 1180 ts_tablet_manager.cc:1918] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.040715 1180 log.cc:1198] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/49042c982df448e2b61ce38de50bf1a1
I20250110 02:51:09.041117 1180 ts_tablet_manager.cc:1939] T 49042c982df448e2b61ce38de50bf1a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.042755 1184 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.042910 1180 tablet_replica.cc:331] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.042906 1184 ts_tablet_manager.cc:1397] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.004s
I20250110 02:51:09.043136 1180 raft_consensus.cc:2238] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.043306 1180 raft_consensus.cc:2267] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.043596 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 49042c982df448e2b61ce38de50bf1a1 (table default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]) successfully deleted
W20250110 02:51:09.043895 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b851ddf27fe44d19b9ca39d82e6352cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b851ddf27fe44d19b9ca39d82e6352cd already in progress: creating tablet
I20250110 02:51:09.043849 1184 raft_consensus.cc:357] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.044028 1184 raft_consensus.cc:383] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:09.044107 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e91197c808ba4710970fcdb57f06d4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e91197c808ba4710970fcdb57f06d4b9 already in progress: creating tablet
I20250110 02:51:09.044139 1184 raft_consensus.cc:738] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.044365 1184 consensus_queue.cc:260] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.044798 1180 ts_tablet_manager.cc:1905] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.046509 1184 ts_tablet_manager.cc:1428] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:09.048066 1184 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.048349 1188 tablet_bootstrap.cc:492] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.048513 1188 ts_tablet_manager.cc:1397] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.020s
I20250110 02:51:09.049301 1188 raft_consensus.cc:357] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.049530 1188 raft_consensus.cc:383] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.049620 1188 raft_consensus.cc:738] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.049638 1184 tablet_bootstrap.cc:654] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.049854 1188 consensus_queue.cc:260] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.050415 1188 ts_tablet_manager.cc:1428] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:09.052683 1179 tablet_replica.cc:331] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.052928 1179 raft_consensus.cc:2238] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:09.053086 1188 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.053155 1179 pending_rounds.cc:62] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:09.053227 1179 pending_rounds.cc:69] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611713042071552 op_type: NO_OP noop_request { }
I20250110 02:51:09.053339 1179 raft_consensus.cc:2883] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:09.053442 1179 raft_consensus.cc:2267] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.055305 1188 tablet_bootstrap.cc:654] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.059336 1181 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.060667 1184 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.060832 1184 ts_tablet_manager.cc:1397] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20250110 02:51:09.061614 1181 tablet_bootstrap.cc:654] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.061712 1184 raft_consensus.cc:357] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.062001 1184 raft_consensus.cc:383] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.062255 1184 raft_consensus.cc:738] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.062527 1184 consensus_queue.cc:260] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.062984 1180 ts_tablet_manager.cc:1918] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.063146 1180 log.cc:1198] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c81e5a619cd643dea7b457f4916ecf68
I20250110 02:51:09.063925 1180 ts_tablet_manager.cc:1939] T c81e5a619cd643dea7b457f4916ecf68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.065213 1184 ts_tablet_manager.cc:1428] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250110 02:51:09.066524 1184 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.069365 897 tablet_service.cc:1467] Processing CreateTablet for tablet 866513b2738844bba19bb91a7f4bfd81 (DEFAULT_TABLE table=default.Table_7670a09cdcf64f9c9c258bcda9f0d69f [id=308c5c775cc34795af11c57c3593cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.069551 1184 tablet_bootstrap.cc:654] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.069928 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866513b2738844bba19bb91a7f4bfd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.071193 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c81e5a619cd643dea7b457f4916ecf68 (table default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]) successfully deleted
I20250110 02:51:09.072556 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 866513b2738844bba19bb91a7f4bfd81 (DEFAULT_TABLE table=default.Table_7670a09cdcf64f9c9c258bcda9f0d69f [id=308c5c775cc34795af11c57c3593cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.072948 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866513b2738844bba19bb91a7f4bfd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.073138 755 tablet_service.cc:1467] Processing CreateTablet for tablet 866513b2738844bba19bb91a7f4bfd81 (DEFAULT_TABLE table=default.Table_7670a09cdcf64f9c9c258bcda9f0d69f [id=308c5c775cc34795af11c57c3593cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.073508 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866513b2738844bba19bb91a7f4bfd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.077489 1184 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.077628 1184 ts_tablet_manager.cc:1397] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250110 02:51:09.078007 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_30e7d7a0bb514870bfae56726686e7be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.078526 1184 raft_consensus.cc:357] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.078747 1184 raft_consensus.cc:383] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.078842 1184 raft_consensus.cc:738] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.079095 1184 consensus_queue.cc:260] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.079326 1181 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
W20250110 02:51:09.079442 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30e7d7a0bb514870bfae56726686e7be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.079509 1181 ts_tablet_manager.cc:1397] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.001s
I20250110 02:51:09.081533 1184 ts_tablet_manager.cc:1428] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:09.082769 1181 raft_consensus.cc:357] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.083086 1181 raft_consensus.cc:383] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.083199 1181 raft_consensus.cc:738] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.083487 1181 consensus_queue.cc:260] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.083786 1184 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.085322 1184 tablet_bootstrap.cc:654] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.086277 1181 ts_tablet_manager.cc:1428] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:09.087606 1181 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.089236 1181 tablet_bootstrap.cc:654] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.090437 1188 tablet_bootstrap.cc:492] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.090598 1188 ts_tablet_manager.cc:1397] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.017s
I20250110 02:51:09.091302 1188 raft_consensus.cc:357] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.091485 1188 raft_consensus.cc:383] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.091573 1188 raft_consensus.cc:738] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.091778 1188 consensus_queue.cc:260] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.092247 1188 ts_tablet_manager.cc:1428] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:09.092542 1188 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.094489 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5381a3601aa745659f017c21643544c5" } new_table_name: "default.Table_faaf96de46b845ffa89aae241b177d29" modify_external_catalogs: true
I20250110 02:51:09.095484 1178 ts_tablet_manager.cc:1918] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.095744 1178 log.cc:1198] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/54fd7ceb38cf413a8b8c6c2e710d4adf
I20250110 02:51:09.096235 1178 ts_tablet_manager.cc:1939] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.097137 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1e5670d3b7e54fde8629a6044c5e4f16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.097393 1181 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.097558 1181 ts_tablet_manager.cc:1397] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
W20250110 02:51:09.097689 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e5670d3b7e54fde8629a6044c5e4f16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.098754 1188 tablet_bootstrap.cc:654] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.099099 1181 raft_consensus.cc:357] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.099295 1181 raft_consensus.cc:383] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.099378 1181 raft_consensus.cc:738] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.099658 1181 consensus_queue.cc:260] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:09.101881 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 286aee6074e14c27b17bc79a78906620 consensus state has no leader
I20250110 02:51:09.104238 1179 ts_tablet_manager.cc:1905] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.104588 1184 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.104773 1184 ts_tablet_manager.cc:1397] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20250110 02:51:09.104972 1188 tablet_bootstrap.cc:492] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.105124 1188 ts_tablet_manager.cc:1397] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20250110 02:51:09.105597 1184 raft_consensus.cc:357] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.105810 1184 raft_consensus.cc:383] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.105901 1184 raft_consensus.cc:738] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.105849 1188 raft_consensus.cc:357] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.106071 1188 raft_consensus.cc:383] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.106155 1188 raft_consensus.cc:738] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.106585 1181 ts_tablet_manager.cc:1428] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20250110 02:51:09.106786 1184 consensus_queue.cc:260] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.106441 1188 consensus_queue.cc:260] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.109095 1181 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.109838 1184 ts_tablet_manager.cc:1428] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:09.110222 521 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]). Expected version 1 got 0
I20250110 02:51:09.110430 1188 ts_tablet_manager.cc:1428] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250110 02:51:09.111310 1181 tablet_bootstrap.cc:654] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.111660 518 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.112501 1188 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.114236 1188 tablet_bootstrap.cc:654] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.114440 518 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.114620 518 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]). Expected version 1 got 0
I20250110 02:51:09.114753 517 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]). Expected version 1 got 0
I20250110 02:51:09.115024 517 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 2 got 0
I20250110 02:51:09.115254 1179 ts_tablet_manager.cc:1918] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:09.115376 1179 log.cc:1198] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/ce7d45ee5df746d99e38a7cfba334c98
I20250110 02:51:09.115772 1179 ts_tablet_manager.cc:1939] T ce7d45ee5df746d99e38a7cfba334c98 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.116133 521 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]). Expected version 1 got 0
I20250110 02:51:09.116917 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet ce7d45ee5df746d99e38a7cfba334c98 (table default.Table_3a960be627b447e0bd193430536685e1 [id=546c12da804742c4b04c80742b972a01]) successfully deleted
I20250110 02:51:09.117565 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:09.117615 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b851ddf27fe44d19b9ca39d82e6352cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b851ddf27fe44d19b9ca39d82e6352cd already in progress: creating tablet
W20250110 02:51:09.117841 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e91197c808ba4710970fcdb57f06d4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e91197c808ba4710970fcdb57f06d4b9 already in progress: creating tablet
I20250110 02:51:09.118726 897 tablet_service.cc:1467] Processing CreateTablet for tablet 9bce8f5888bb404696e46487176b5fe8 (DEFAULT_TABLE table=default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.118806 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 9bce8f5888bb404696e46487176b5fe8 (DEFAULT_TABLE table=default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.119109 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bce8f5888bb404696e46487176b5fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.119144 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bce8f5888bb404696e46487176b5fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.120863 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd7ceb38cf413a8b8c6c2e710d4adf with delete_type TABLET_DATA_DELETED (replaced by tablet e91197c808ba4710970fcdb57f06d4b9) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.121112 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e8541e1c6d454877855c6b1b2e6e334f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.121331 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 36fa58b399864d678c81e6bd0d43e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.121520 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7d45ee5df746d99e38a7cfba334c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.122020 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f34d917babd4b4e8834260fa5dacba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.122530 1179 tablet_replica.cc:331] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.122855 1179 raft_consensus.cc:2238] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.123070 1179 raft_consensus.cc:2267] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.123329 517 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.123519 517 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]). Expected version 1 got 0
I20250110 02:51:09.123948 1188 tablet_bootstrap.cc:492] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.124091 1188 ts_tablet_manager.cc:1397] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20250110 02:51:09.124171 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_868905ee96fe4989884cf84f8711954e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.124666 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_868905ee96fe4989884cf84f8711954e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.124878 1188 raft_consensus.cc:357] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.125078 1188 raft_consensus.cc:383] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.125088 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.125181 1188 raft_consensus.cc:738] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.125411 1188 consensus_queue.cc:260] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.125757 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c81e5a619cd643dea7b457f4916ecf68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.126058 1188 ts_tablet_manager.cc:1428] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:09.126307 1188 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.126785 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9bce8f5888bb404696e46487176b5fe8 (DEFAULT_TABLE table=default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.126933 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 4c845c27a86d4e4f93c932313ebf648b (DEFAULT_TABLE table=default.Table_6b27d5e1d8994346acba31f9770550b0 [id=2f13169995da4860913a0bf7455da3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.127144 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bce8f5888bb404696e46487176b5fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.127256 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c845c27a86d4e4f93c932313ebf648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.128899 1188 tablet_bootstrap.cc:654] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.130852 755 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.132606 518 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]). Expected version 1 got 0
I20250110 02:51:09.133814 897 tablet_service.cc:1467] Processing CreateTablet for tablet 4c845c27a86d4e4f93c932313ebf648b (DEFAULT_TABLE table=default.Table_6b27d5e1d8994346acba31f9770550b0 [id=2f13169995da4860913a0bf7455da3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.134192 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c845c27a86d4e4f93c932313ebf648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.139664 896 tablet_service.cc:1467] Processing CreateTablet for tablet 886ab6d12cf545b9b700959e18396c50 (DEFAULT_TABLE table=default.Table_b164fa0aebbe4bb2821b21038f738d0c [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.139995 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886ab6d12cf545b9b700959e18396c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.141754 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4c845c27a86d4e4f93c932313ebf648b (DEFAULT_TABLE table=default.Table_6b27d5e1d8994346acba31f9770550b0 [id=2f13169995da4860913a0bf7455da3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.142114 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c845c27a86d4e4f93c932313ebf648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.145697 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1e5670d3b7e54fde8629a6044c5e4f16" } modify_external_catalogs: true
I20250110 02:51:09.145861 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1e5670d3b7e54fde8629a6044c5e4f16" } modify_external_catalogs: true
I20250110 02:51:09.146579 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f4f45d5a3c9748859376185f7b09115d" } modify_external_catalogs: true
I20250110 02:51:09.146713 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f4f45d5a3c9748859376185f7b09115d" } modify_external_catalogs: true
I20250110 02:51:09.147450 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 886ab6d12cf545b9b700959e18396c50 (DEFAULT_TABLE table=default.Table_b164fa0aebbe4bb2821b21038f738d0c [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.147818 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886ab6d12cf545b9b700959e18396c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.150457 749 tablet_service.cc:1467] Processing CreateTablet for tablet 886ab6d12cf545b9b700959e18396c50 (DEFAULT_TABLE table=default.Table_b164fa0aebbe4bb2821b21038f738d0c [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.150825 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886ab6d12cf545b9b700959e18396c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.151403 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_50073d844fb342e19b52d0914b56098f" } new_table_name: "default.Table_4f862664cf4044afbe096ffdf7960b32" modify_external_catalogs: true
I20250110 02:51:09.155016 1184 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.155972 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ad166793c0b34c8bb6bceca5b8580839" } new_table_name: "default.Table_cbaca8b1d598406db88d9fcef46dd17b" modify_external_catalogs: true
I20250110 02:51:09.160987 523 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 918fe7a7d1dc404fbf50c281ec84429f
I20250110 02:51:09.162879 749 tablet_service.cc:1467] Processing CreateTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.163213 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f6f4f221e6445db0bb569f372cfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.163409 1184 tablet_bootstrap.cc:654] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.165020 1179 ts_tablet_manager.cc:1905] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.166493 519 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8e31e380160f40f4a7ced9e3999efdc3
W20250110 02:51:09.169821 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10504e77a5c64ebf85be937d64ca9c80 consensus state has no leader
I20250110 02:51:09.171288 896 tablet_service.cc:1467] Processing CreateTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.171643 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f6f4f221e6445db0bb569f372cfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.176124 521 master_service.cc:946] ReplaceTablet: received request to replace tablet b85eea2ad540445dbd95a655135dc043 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.176421 1181 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.176587 1181 ts_tablet_manager.cc:1397] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20250110 02:51:09.177173 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.178350 1178 tablet_replica.cc:331] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.178583 1178 raft_consensus.cc:2238] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.178817 1178 raft_consensus.cc:2267] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.179437 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 (DEFAULT_TABLE table=default.Table_fb098e79694d4feaafc62a7c3d9eef8d [id=812b6d95174a4614857cae88273885cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.179819 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66f6f4f221e6445db0bb569f372cfaa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.180116 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b (DEFAULT_TABLE table=default.Table_ad166793c0b34c8bb6bceca5b8580839 [id=b74f16dd5b58413c8dff82451dd83cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.180437 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a27f0327d1844b8a561af8b0bf4385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.180611 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f8b1849990d84955a6b754fdf58919d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.181077 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b1849990d84955a6b754fdf58919d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.182107 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_30e7d7a0bb514870bfae56726686e7be" } new_table_name: "default.Table_88a8376ca7864049a601575865aa9b88" modify_external_catalogs: true
I20250110 02:51:09.184772 1178 ts_tablet_manager.cc:1905] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.179013 1181 raft_consensus.cc:357] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.185889 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b (DEFAULT_TABLE table=default.Table_ad166793c0b34c8bb6bceca5b8580839 [id=b74f16dd5b58413c8dff82451dd83cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.186286 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a27f0327d1844b8a561af8b0bf4385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.186465 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_69fd953800254930800f1ff2d83a8907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.187498 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b (DEFAULT_TABLE table=default.Table_d4d87a97dad24a4784c3a82e4bf288f9 [id=bee875a9b8b2438db4aecd5f82a5044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.187804 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d5f9bda55942ff856252bbb50b3c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.191210 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_fb098e79694d4feaafc62a7c3d9eef8d" } new_table_name: "default.Table_046451c051e2479b8b7de432ffb408bc" modify_external_catalogs: true
W20250110 02:51:09.192018 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69fd953800254930800f1ff2d83a8907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.192386 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a1233829553f48db897d4e93aa37c38d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.192849 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1233829553f48db897d4e93aa37c38d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.192963 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.193336 519 catalog_manager.cc:6596] ReplaceTablet: tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 deleted and replaced by tablet a9f404a2e7494645af449aea3d3d3bf0
I20250110 02:51:09.193838 517 master_service.cc:946] ReplaceTablet: received request to replace tablet e91197c808ba4710970fcdb57f06d4b9 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.194015 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f404a2e7494645af449aea3d3d3bf0) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:09.194165 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a27f0327d1844b8a561af8b0bf4385b consensus state has no leader
I20250110 02:51:09.198552 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f404a2e7494645af449aea3d3d3bf0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.198791 1180 tablet_replica.cc:331] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.198958 1180 raft_consensus.cc:2238] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.199134 1180 raft_consensus.cc:2267] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.199625 1180 ts_tablet_manager.cc:1905] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.201596 1187 raft_consensus.cc:491] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.201768 1187 raft_consensus.cc:513] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.202497 1187 leader_election.cc:290] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:09.202719 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67adef4c3f4041e1ab480100ca762f62 has no consensus state
I20250110 02:51:09.203145 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c46be1162124b50ae43d8ea57cb5a0b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.203331 1061 raft_consensus.cc:2463] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.203709 711 leader_election.cc:304] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.203959 1187 raft_consensus.cc:2798] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.204026 1187 raft_consensus.cc:491] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.204099 1187 raft_consensus.cc:3054] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.204767 1180 ts_tablet_manager.cc:1918] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.204877 1180 log.cc:1198] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.205215 1180 ts_tablet_manager.cc:1939] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.206041 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:09.206543 1214 raft_consensus.cc:491] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.206604 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b (DEFAULT_TABLE table=default.Table_ad166793c0b34c8bb6bceca5b8580839 [id=b74f16dd5b58413c8dff82451dd83cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.206689 1214 raft_consensus.cc:513] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.207054 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a27f0327d1844b8a561af8b0bf4385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.207278 1214 leader_election.cc:290] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.209754 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 (DEFAULT_TABLE table=default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.210052 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 (DEFAULT_TABLE table=default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.210160 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aafff2e0de548129d99e9f07dc3ab11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.210685 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aafff2e0de548129d99e9f07dc3ab11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.211251 1184 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.211375 1184 ts_tablet_manager.cc:1397] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.003s
I20250110 02:51:09.212126 1184 raft_consensus.cc:357] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.212291 1184 raft_consensus.cc:383] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.212388 1184 raft_consensus.cc:738] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.212636 1184 consensus_queue.cc:260] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.214443 1181 raft_consensus.cc:383] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:09.214458 711 leader_election.cc:343] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.214540 1181 raft_consensus.cc:738] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.214782 1181 consensus_queue.cc:260] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.216199 1184 ts_tablet_manager.cc:1428] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250110 02:51:09.216476 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.216888 1181 ts_tablet_manager.cc:1428] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250110 02:51:09.217180 1184 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.218269 1181 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.219637 1184 tablet_bootstrap.cc:654] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.221557 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.221776 1180 tablet_replica.cc:331] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.221941 1180 raft_consensus.cc:2238] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.222079 1180 raft_consensus.cc:2267] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.223191 1180 ts_tablet_manager.cc:1905] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.223527 1181 tablet_bootstrap.cc:654] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.224257 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f404a2e7494645af449aea3d3d3bf0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.225687 1179 ts_tablet_manager.cc:1918] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.225813 1179 log.cc:1198] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/54fd7ceb38cf413a8b8c6c2e710d4adf
W20250110 02:51:09.226089 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66f6f4f221e6445db0bb569f372cfaa0 consensus state has no leader
I20250110 02:51:09.226150 1179 ts_tablet_manager.cc:1939] T 54fd7ceb38cf413a8b8c6c2e710d4adf P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.232111 1187 raft_consensus.cc:513] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.233279 1187 leader_election.cc:290] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.233508 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c46be1162124b50ae43d8ea57cb5a0b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.233685 1061 raft_consensus.cc:3054] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.235389 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cbaca8b1d598406db88d9fcef46dd17b" } modify_external_catalogs: true
I20250110 02:51:09.235592 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cbaca8b1d598406db88d9fcef46dd17b" } modify_external_catalogs: true
I20250110 02:51:09.236081 1061 raft_consensus.cc:2463] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.236629 711 leader_election.cc:304] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.237617 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f97a80832b8348cc9f5839ebe6887044" } modify_external_catalogs: true
I20250110 02:51:09.237803 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f97a80832b8348cc9f5839ebe6887044" } modify_external_catalogs: true
I20250110 02:51:09.238206 1187 raft_consensus.cc:2798] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.239195 1187 raft_consensus.cc:695] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.239670 1187 consensus_queue.cc:237] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.241024 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_69fd953800254930800f1ff2d83a8907" } modify_external_catalogs: true
I20250110 02:51:09.241283 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_69fd953800254930800f1ff2d83a8907" } modify_external_catalogs: true
I20250110 02:51:09.230348 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c46be1162124b50ae43d8ea57cb5a0b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.242517 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7059bcdf7e4d4cf4b559b00baf58f6c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.244271 917 raft_consensus.cc:2463] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.244534 918 raft_consensus.cc:2463] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.246698 755 tablet_service.cc:1467] Processing CreateTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b (DEFAULT_TABLE table=default.Table_d4d87a97dad24a4784c3a82e4bf288f9 [id=bee875a9b8b2438db4aecd5f82a5044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.246773 897 tablet_service.cc:1467] Processing CreateTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 (DEFAULT_TABLE table=default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.247082 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d5f9bda55942ff856252bbb50b3c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.248944 894 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.249342 891 tablet_service.cc:1467] Processing CreateTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.249492 750 tablet_service.cc:1467] Processing CreateTablet for tablet c0edfa33e30042edad853217aa761bd5 (DEFAULT_TABLE table=default.Table_f8b1849990d84955a6b754fdf58919d4 [id=4f08fffc4d93445687c1abf05f669666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.249573 756 tablet_service.cc:1467] Processing CreateTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.249958 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f404a2e7494645af449aea3d3d3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.250181 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0edfa33e30042edad853217aa761bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.250238 898 tablet_service.cc:1467] Processing CreateTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b (DEFAULT_TABLE table=default.Table_d4d87a97dad24a4784c3a82e4bf288f9 [id=bee875a9b8b2438db4aecd5f82a5044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.250494 752 tablet_service.cc:1467] Processing CreateTablet for tablet 67adef4c3f4041e1ab480100ca762f62 (DEFAULT_TABLE table=default.Table_88a8376ca7864049a601575865aa9b88 [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.250626 895 tablet_service.cc:1467] Processing CreateTablet for tablet c0edfa33e30042edad853217aa761bd5 (DEFAULT_TABLE table=default.Table_f8b1849990d84955a6b754fdf58919d4 [id=4f08fffc4d93445687c1abf05f669666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.250859 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67adef4c3f4041e1ab480100ca762f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.251196 893 tablet_service.cc:1467] Processing CreateTablet for tablet 67adef4c3f4041e1ab480100ca762f62 (DEFAULT_TABLE table=default.Table_88a8376ca7864049a601575865aa9b88 [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.251441 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c46be1162124b50ae43d8ea57cb5a0b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.251626 918 raft_consensus.cc:3054] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.252343 709 leader_election.cc:304] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:09.252972 1214 raft_consensus.cc:2798] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.253084 1214 raft_consensus.cc:491] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.253192 1214 raft_consensus.cc:3054] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.253979 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.254312 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d791b1c504171ba66dec55c8fb0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.254707 1184 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.254844 1184 ts_tablet_manager.cc:1397] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.009s
I20250110 02:51:09.254206 1181 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.255270 1181 ts_tablet_manager.cc:1397] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.003s
I20250110 02:51:09.255591 1184 raft_consensus.cc:357] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.255827 1184 raft_consensus.cc:383] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.255903 1184 raft_consensus.cc:738] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.256000 1181 raft_consensus.cc:357] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.256227 1181 raft_consensus.cc:383] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.256314 1181 raft_consensus.cc:738] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.256565 1181 consensus_queue.cc:260] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.257828 1184 consensus_queue.cc:260] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.258858 1184 ts_tablet_manager.cc:1428] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250110 02:51:09.259148 1184 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.259753 1181 ts_tablet_manager.cc:1428] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:09.260680 1184 tablet_bootstrap.cc:654] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.260717 1181 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.262482 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aafff2e0de548129d99e9f07dc3ab11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.262739 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.263079 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d791b1c504171ba66dec55c8fb0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.263159 1040 tablet_service.cc:1467] Processing CreateTablet for tablet c0edfa33e30042edad853217aa761bd5 (DEFAULT_TABLE table=default.Table_f8b1849990d84955a6b754fdf58919d4 [id=4f08fffc4d93445687c1abf05f669666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.263624 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0edfa33e30042edad853217aa761bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.266534 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f404a2e7494645af449aea3d3d3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.266728 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 67adef4c3f4041e1ab480100ca762f62 (DEFAULT_TABLE table=default.Table_88a8376ca7864049a601575865aa9b88 [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.267129 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67adef4c3f4041e1ab480100ca762f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.269090 1033 tablet_service.cc:1467] Processing CreateTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.269383 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f404a2e7494645af449aea3d3d3bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.270988 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d791b1c504171ba66dec55c8fb0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.274591 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d5f9bda55942ff856252bbb50b3c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.275985 1181 tablet_bootstrap.cc:654] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.278316 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67adef4c3f4041e1ab480100ca762f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.278389 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.285638 918 raft_consensus.cc:2463] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.285862 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0edfa33e30042edad853217aa761bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.288477 1181 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.288628 1181 ts_tablet_manager.cc:1397] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.004s
I20250110 02:51:09.289398 1181 raft_consensus.cc:357] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.289613 1181 raft_consensus.cc:383] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.289685 1181 raft_consensus.cc:738] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.289927 1181 consensus_queue.cc:260] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.291563 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b691c5c8267948a08b5a76dc5675aa52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.291824 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b09e01dad0d447bfbbcac8794a1c9cff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.292200 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b691c5c8267948a08b5a76dc5675aa52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:09.292263 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b09e01dad0d447bfbbcac8794a1c9cff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.292906 1181 ts_tablet_manager.cc:1428] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:09.295444 1181 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.297122 1181 tablet_bootstrap.cc:654] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.299736 1188 tablet_bootstrap.cc:492] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.299618 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_009422f290cd4fc9983b13a098c7b13b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.300135 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_009422f290cd4fc9983b13a098c7b13b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.300343 1188 ts_tablet_manager.cc:1397] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.174s user 0.026s sys 0.042s
I20250110 02:51:09.301169 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39025
--webserver_interface=127.0.104.60
--webserver_port=34089
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 02:51:09.301251 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b85eea2ad540445dbd95a655135dc043
I20250110 02:51:09.302067 521 catalog_manager.cc:6596] ReplaceTablet: tablet b85eea2ad540445dbd95a655135dc043 deleted and replaced by tablet 2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:09.302812 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b85eea2ad540445dbd95a655135dc043 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da6483de7f5475ea3d755a6efbda369) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.303736 1181 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.303876 1181 ts_tablet_manager.cc:1397] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:09.304716 1181 raft_consensus.cc:357] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.304930 1181 raft_consensus.cc:383] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.305014 1181 raft_consensus.cc:738] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.305286 1181 consensus_queue.cc:260] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.307422 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5f6d11dc852b4cd189790829bca600ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.307956 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f6d11dc852b4cd189790829bca600ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.308822 1181 ts_tablet_manager.cc:1428] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:09.308861 525 master_service.cc:946] ReplaceTablet: received request to replace tablet b851ddf27fe44d19b9ca39d82e6352cd from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.310084 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b85eea2ad540445dbd95a655135dc043 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da6483de7f5475ea3d755a6efbda369) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.311100 1188 raft_consensus.cc:357] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.311460 1188 raft_consensus.cc:383] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.311543 1188 raft_consensus.cc:738] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.311923 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.312173 1188 consensus_queue.cc:260] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.312515 1181 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.312721 1188 ts_tablet_manager.cc:1428] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250110 02:51:09.313253 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b85eea2ad540445dbd95a655135dc043 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da6483de7f5475ea3d755a6efbda369) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.312294 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_627ff2f24b97485a9f948b72f8b39daf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.313431 1188 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:09.313652 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_627ff2f24b97485a9f948b72f8b39daf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.314399 1181 tablet_bootstrap.cc:654] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.314944 1188 tablet_bootstrap.cc:654] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.322144 1180 ts_tablet_manager.cc:1918] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.322432 523 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 983a71f3cb9d48c2ba3d50e7606650f3
I20250110 02:51:09.326339 1178 ts_tablet_manager.cc:1918] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.326476 1178 log.cc:1198] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e8541e1c6d454877855c6b1b2e6e334f
I20250110 02:51:09.326838 1178 ts_tablet_manager.cc:1939] T e8541e1c6d454877855c6b1b2e6e334f P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.327761 1178 tablet_replica.cc:331] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.327939 1178 raft_consensus.cc:2238] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.328151 1178 raft_consensus.cc:2267] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.328477 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e8541e1c6d454877855c6b1b2e6e334f (table default.Table_35ffcbc725fc49f0b84935fa73df1a75 [id=c13502bad83548af8d88648501d67b46]) successfully deleted
I20250110 02:51:09.328917 1178 ts_tablet_manager.cc:1905] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.329130 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1c46be1162124b50ae43d8ea57cb5a0b
I20250110 02:51:09.330036 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.332499 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.332981 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e44620c0b9ce431ba206f9111c076233" } new_table_name: "default.Table_f0abf378d8b94be9b42028b3ef26bbe5" modify_external_catalogs: true
I20250110 02:51:09.325837 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 49042c982df448e2b61ce38de50bf1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.334532 1178 ts_tablet_manager.cc:1918] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.334753 1178 log.cc:1198] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/36fa58b399864d678c81e6bd0d43e790
I20250110 02:51:09.335098 1178 ts_tablet_manager.cc:1939] T 36fa58b399864d678c81e6bd0d43e790 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.335755 1179 tablet_replica.cc:331] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.335987 1179 raft_consensus.cc:2238] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.336169 1179 raft_consensus.cc:2267] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.336836 1179 ts_tablet_manager.cc:1905] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.336829 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 36fa58b399864d678c81e6bd0d43e790 (table default.Table_3786788ec1f94ce7b71c09c11f0d5ebe [id=3ce7a84313d14aef98dda1679fb87f18]) successfully deleted
I20250110 02:51:09.337188 1178 tablet_replica.cc:331] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.337407 1178 raft_consensus.cc:2238] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.337533 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 54fd7ceb38cf413a8b8c6c2e710d4adf (table default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]) successfully deleted
I20250110 02:51:09.337611 1178 raft_consensus.cc:2267] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.338184 1178 ts_tablet_manager.cc:1905] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:09.338333 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e8541e1c6d454877855c6b1b2e6e334f because the tablet was not found. No further retry: Not found: Tablet not found: e8541e1c6d454877855c6b1b2e6e334f
W20250110 02:51:09.339053 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 36fa58b399864d678c81e6bd0d43e790 because the tablet was not found. No further retry: Not found: Tablet not found: 36fa58b399864d678c81e6bd0d43e790
W20250110 02:51:09.339510 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet ce7d45ee5df746d99e38a7cfba334c98 because the tablet was not found. No further retry: Not found: Tablet not found: ce7d45ee5df746d99e38a7cfba334c98
W20250110 02:51:09.339954 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7f34d917babd4b4e8834260fa5dacba4 because the tablet was not found. No further retry: Not found: Tablet not found: 7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:09.338440 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8da521fca5664959b1c0801b30199a6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.341611 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8da521fca5664959b1c0801b30199a6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.343958 1178 ts_tablet_manager.cc:1918] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.344135 1178 log.cc:1198] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/28ba5987d51a433e886909dffdc3ef8e
I20250110 02:51:09.344545 1178 ts_tablet_manager.cc:1939] T 28ba5987d51a433e886909dffdc3ef8e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.345511 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 28ba5987d51a433e886909dffdc3ef8e (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
I20250110 02:51:09.345867 1178 tablet_replica.cc:331] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.346145 1178 raft_consensus.cc:2238] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.346406 1178 raft_consensus.cc:2267] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.347086 1178 ts_tablet_manager.cc:1905] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.347919 1214 raft_consensus.cc:513] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.350267 1179 ts_tablet_manager.cc:1918] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.350395 1179 log.cc:1198] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/49042c982df448e2b61ce38de50bf1a1
I20250110 02:51:09.350745 1179 ts_tablet_manager.cc:1939] T 49042c982df448e2b61ce38de50bf1a1 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.351588 1179 tablet_replica.cc:331] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.351788 1179 raft_consensus.cc:2238] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.351927 1179 raft_consensus.cc:2267] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.352144 1180 log.cc:1198] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:09.352449 1180 ts_tablet_manager.cc:1939] T fe17b2fbe80d4468adf9adb20fd86676 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.353394 1180 tablet_replica.cc:331] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.353557 1180 raft_consensus.cc:2238] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.353765 1180 raft_consensus.cc:2267] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.330670 1181 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.337868 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.354048 1181 ts_tablet_manager.cc:1397] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20250110 02:51:09.354249 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7059bcdf7e4d4cf4b559b00baf58f6c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.354473 918 raft_consensus.cc:3054] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.354554 1188 tablet_bootstrap.cc:492] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.354664 1188 ts_tablet_manager.cc:1397] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.004s
I20250110 02:51:09.354725 1214 leader_election.cc:290] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.354918 1181 raft_consensus.cc:357] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.355113 1181 raft_consensus.cc:383] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:09.355088 711 leader_election.cc:343] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.355170 1181 raft_consensus.cc:738] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.355379 1188 raft_consensus.cc:357] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.355404 1181 consensus_queue.cc:260] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.355569 1188 raft_consensus.cc:383] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.355664 1188 raft_consensus.cc:738] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.356225 1188 consensus_queue.cc:260] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.357887 918 raft_consensus.cc:2463] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.358309 709 leader_election.cc:304] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:09.358829 1214 raft_consensus.cc:2798] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.359066 1184 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.359195 1184 ts_tablet_manager.cc:1397] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.100s user 0.020s sys 0.014s
I20250110 02:51:09.357934 1188 ts_tablet_manager.cc:1428] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:09.360105 1214 raft_consensus.cc:695] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.360253 1184 raft_consensus.cc:357] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.360390 1214 consensus_queue.cc:237] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.360518 1184 raft_consensus.cc:383] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.360824 1188 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.338776 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c46be1162124b50ae43d8ea57cb5a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.362341 1183 raft_consensus.cc:491] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.362468 1188 tablet_bootstrap.cc:654] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.362499 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b09e01dad0d447bfbbcac8794a1c9cff" } modify_external_catalogs: true
I20250110 02:51:09.362522 1183 raft_consensus.cc:513] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.362676 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b09e01dad0d447bfbbcac8794a1c9cff" } modify_external_catalogs: true
I20250110 02:51:09.363021 1178 ts_tablet_manager.cc:1918] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.363152 1178 log.cc:1198] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7f34d917babd4b4e8834260fa5dacba4
I20250110 02:51:09.363781 1178 ts_tablet_manager.cc:1939] T 7f34d917babd4b4e8834260fa5dacba4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.364113 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet fe17b2fbe80d4468adf9adb20fd86676 (table default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]) successfully deleted
I20250110 02:51:09.364288 1183 leader_election.cc:290] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
W20250110 02:51:09.364408 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b85eea2ad540445dbd95a655135dc043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85eea2ad540445dbd95a655135dc043 already in progress: creating tablet
I20250110 02:51:09.364634 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7f34d917babd4b4e8834260fa5dacba4 (table default.Table_1f2bd93f6ba1455daeaef5046baefe0b [id=eac2742a551d4349bb9ef71b43f7e1aa]) successfully deleted
I20250110 02:51:09.364861 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 49042c982df448e2b61ce38de50bf1a1 (table default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]) successfully deleted
W20250110 02:51:09.365567 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b851ddf27fe44d19b9ca39d82e6352cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b851ddf27fe44d19b9ca39d82e6352cd already in progress: creating tablet
W20250110 02:51:09.366282 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e91197c808ba4710970fcdb57f06d4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e91197c808ba4710970fcdb57f06d4b9 already in progress: creating tablet
I20250110 02:51:09.366639 1178 tablet_replica.cc:331] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.366647 1184 raft_consensus.cc:738] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.366928 1178 raft_consensus.cc:2238] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.367142 1178 raft_consensus.cc:2267] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.367131 1184 consensus_queue.cc:260] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.367753 1178 ts_tablet_manager.cc:1905] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.370213 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_46bcf45f1fce419c882b8c239e680ee9" } modify_external_catalogs: true
I20250110 02:51:09.370417 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_46bcf45f1fce419c882b8c239e680ee9" } modify_external_catalogs: true
I20250110 02:51:09.371188 1184 ts_tablet_manager.cc:1428] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250110 02:51:09.372805 1184 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.373411 1178 ts_tablet_manager.cc:1918] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.373522 1178 log.cc:1198] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c81e5a619cd643dea7b457f4916ecf68
I20250110 02:51:09.373853 1178 ts_tablet_manager.cc:1939] T c81e5a619cd643dea7b457f4916ecf68 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.375134 1184 tablet_bootstrap.cc:654] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.375357 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:09.375633 776 raft_consensus.cc:2463] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:09.376322 996 leader_election.cc:304] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.376883 1183 raft_consensus.cc:2798] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.377012 1183 raft_consensus.cc:491] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.377106 1183 raft_consensus.cc:3054] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.379194 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c81e5a619cd643dea7b457f4916ecf68 (table default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]) successfully deleted
I20250110 02:51:09.380342 1178 tablet_replica.cc:331] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.380801 1178 raft_consensus.cc:2238] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.380846 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7377066199724b26af78c183f9811324"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.381386 1178 raft_consensus.cc:2267] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:09.381439 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7377066199724b26af78c183f9811324 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.382242 1188 tablet_bootstrap.cc:492] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.382474 1184 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.382457 1188 ts_tablet_manager.cc:1397] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20250110 02:51:09.382619 1184 ts_tablet_manager.cc:1397] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250110 02:51:09.383301 1184 raft_consensus.cc:357] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.383299 1188 raft_consensus.cc:357] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.383505 1184 raft_consensus.cc:383] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.383587 1184 raft_consensus.cc:738] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.383601 1188 raft_consensus.cc:383] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.383757 1188 raft_consensus.cc:738] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.383868 1184 consensus_queue.cc:260] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.384018 1188 consensus_queue.cc:260] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.385454 1184 ts_tablet_manager.cc:1428] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:09.385622 1188 ts_tablet_manager.cc:1428] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:09.387421 1188 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:09.387637 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d58e92b9789e49f6bbfbefe3f8f3a0ef consensus state has no leader
I20250110 02:51:09.388062 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7670a09cdcf64f9c9c258bcda9f0d69f" } new_table_name: "default.Table_9c251b39e0f04bf7916d2ce12652063c" modify_external_catalogs: true
I20250110 02:51:09.389861 1184 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.389905 1178 ts_tablet_manager.cc:1905] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.380916 1183 raft_consensus.cc:513] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.391263 1183 leader_election.cc:290] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:09.392076 526 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 870625a74a94475ca8824a16c00d78b5
I20250110 02:51:09.391036 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:09.393029 776 raft_consensus.cc:3054] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.393158 1184 tablet_bootstrap.cc:654] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.396162 1188 tablet_bootstrap.cc:654] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.396270 1181 ts_tablet_manager.cc:1428] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250110 02:51:09.396781 1181 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.398510 1178 ts_tablet_manager.cc:1918] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.399104 1178 log.cc:1198] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.399451 776 raft_consensus.cc:2463] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:09.399734 1178 ts_tablet_manager.cc:1939] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.400841 996 leader_election.cc:304] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.401394 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:09.402017 1184 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.402172 1184 ts_tablet_manager.cc:1397] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20250110 02:51:09.402505 1183 raft_consensus.cc:2798] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:09.403168 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b85eea2ad540445dbd95a655135dc043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85eea2ad540445dbd95a655135dc043 already in progress: creating tablet
I20250110 02:51:09.403074 1184 raft_consensus.cc:357] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.403261 1184 raft_consensus.cc:383] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.403337 1184 raft_consensus.cc:738] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.403602 1184 consensus_queue.cc:260] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.403757 1178 tablet_replica.cc:331] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.404381 1178 raft_consensus.cc:2238] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.404590 521 master_service.cc:946] ReplaceTablet: received request to replace tablet a9f404a2e7494645af449aea3d3d3bf0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.404836 1178 raft_consensus.cc:2267] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.405040 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_faaf96de46b845ffa89aae241b177d29" } new_table_name: "default.Table_ced8b781ad0f4d948f737825145e4dbf" modify_external_catalogs: true
I20250110 02:51:09.405277 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 286aee6074e14c27b17bc79a78906620 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.405536 1188 tablet_bootstrap.cc:492] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.405807 1188 ts_tablet_manager.cc:1397] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20250110 02:51:09.406525 1188 raft_consensus.cc:357] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.406744 1188 raft_consensus.cc:383] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.406832 1188 raft_consensus.cc:738] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.407009 1188 consensus_queue.cc:260] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.407414 1184 ts_tablet_manager.cc:1428] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250110 02:51:09.408470 1184 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.408550 1181 tablet_bootstrap.cc:654] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.410039 1184 tablet_bootstrap.cc:654] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.410702 1188 ts_tablet_manager.cc:1428] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:09.411114 519 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.411808 1188 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:09.412014 512 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 10504e77a5c64ebf85be937d64ca9c80 (table default.Table_4f862664cf4044afbe096ffdf7960b32 [id=dd4008ea14ad40c68b3e71fac9e2be88], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 10504e77a5c64ebf85be937d64ca9c80 consensus state has no leader
I20250110 02:51:09.413194 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
W20250110 02:51:09.413686 523 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 80900124f93d43a0abf93175b16225a7 consensus state has no leader
I20250110 02:51:09.413872 1178 ts_tablet_manager.cc:1905] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.414755 526 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]). Expected version 1 got 0
I20250110 02:51:09.415104 893 tablet_service.cc:1467] Processing CreateTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e (DEFAULT_TABLE table=default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.415477 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ab917bde534e3fbdf5dbae4af9304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.415797 1214 raft_consensus.cc:491] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.415948 519 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
I20250110 02:51:09.415920 1214 raft_consensus.cc:513] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.416527 1214 leader_election.cc:290] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.417214 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91197c808ba4710970fcdb57f06d4b9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:09.417783 709 leader_election.cc:343] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.414671 1188 tablet_bootstrap.cc:654] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.419132 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91197c808ba4710970fcdb57f06d4b9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:09.419716 711 leader_election.cc:343] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.419732 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2da6483de7f5475ea3d755a6efbda369 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.419837 711 leader_election.cc:304] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:09.420069 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6483de7f5475ea3d755a6efbda369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.420253 1214 raft_consensus.cc:2743] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:09.421391 523 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
I20250110 02:51:09.422693 1178 ts_tablet_manager.cc:1918] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.423135 1184 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.423296 1184 ts_tablet_manager.cc:1397] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250110 02:51:09.424183 1184 raft_consensus.cc:357] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.424440 1179 ts_tablet_manager.cc:1905] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.424731 1178 log.cc:1198] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1c46be1162124b50ae43d8ea57cb5a0b
I20250110 02:51:09.425076 1178 ts_tablet_manager.cc:1939] T 1c46be1162124b50ae43d8ea57cb5a0b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.425820 1184 raft_consensus.cc:383] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.425864 1178 tablet_replica.cc:331] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.425927 1184 raft_consensus.cc:738] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.425983 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]) successfully deleted
I20250110 02:51:09.426050 1178 raft_consensus.cc:2238] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.426223 1178 raft_consensus.cc:2267] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.426177 1184 consensus_queue.cc:260] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.426687 1184 ts_tablet_manager.cc:1428] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:09.426964 1184 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.428505 1188 tablet_bootstrap.cc:492] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.428682 1184 tablet_bootstrap.cc:654] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.428674 1188 ts_tablet_manager.cc:1397] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250110 02:51:09.429385 1188 raft_consensus.cc:357] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.429612 1188 raft_consensus.cc:383] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.429695 1188 raft_consensus.cc:738] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.429874 1188 consensus_queue.cc:260] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.431248 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6b27d5e1d8994346acba31f9770550b0" } new_table_name: "default.Table_33f285fec4b94dc38471bf0dbf5fb92b" modify_external_catalogs: true
I20250110 02:51:09.431607 1188 ts_tablet_manager.cc:1428] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:09.431994 1188 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.432447 1214 raft_consensus.cc:491] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.432565 1214 raft_consensus.cc:513] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.433122 1214 leader_election.cc:290] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.433727 1188 tablet_bootstrap.cc:654] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.433684 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f307f4200409aaa47df55f20472d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.433876 918 raft_consensus.cc:2463] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.434322 709 leader_election.cc:304] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.434482 1181 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.434605 1181 ts_tablet_manager.cc:1397] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250110 02:51:09.434624 1214 raft_consensus.cc:2798] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.434712 1214 raft_consensus.cc:491] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.434784 1214 raft_consensus.cc:3054] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.435308 1181 raft_consensus.cc:357] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.435483 1181 raft_consensus.cc:383] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.435545 1181 raft_consensus.cc:738] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.435606 1178 ts_tablet_manager.cc:1905] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.435722 1181 consensus_queue.cc:260] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.437124 1214 raft_consensus.cc:513] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.438055 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f307f4200409aaa47df55f20472d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.438205 918 raft_consensus.cc:3054] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.440707 1179 ts_tablet_manager.cc:1918] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.440810 1179 log.cc:1198] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c81e5a619cd643dea7b457f4916ecf68
I20250110 02:51:09.440783 918 raft_consensus.cc:2463] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.441123 1179 ts_tablet_manager.cc:1939] T c81e5a619cd643dea7b457f4916ecf68 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.441259 709 leader_election.cc:304] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.441581 1187 raft_consensus.cc:2798] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.441718 1187 raft_consensus.cc:695] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.441984 1187 consensus_queue.cc:237] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.442677 1188 tablet_bootstrap.cc:492] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.442786 1188 ts_tablet_manager.cc:1397] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:09.443471 1188 raft_consensus.cc:357] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.443706 1188 raft_consensus.cc:383] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.443833 1188 raft_consensus.cc:738] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.444101 1188 consensus_queue.cc:260] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.444193 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f307f4200409aaa47df55f20472d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.444388 1061 raft_consensus.cc:2463] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.445250 1188 ts_tablet_manager.cc:1428] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:09.446233 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2500bc74e2d44275bab206e0ba400d75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.446355 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e (DEFAULT_TABLE table=default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.446615 1188 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:09.446791 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2500bc74e2d44275bab206e0ba400d75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.446822 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ab917bde534e3fbdf5dbae4af9304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.449599 1178 ts_tablet_manager.cc:1918] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.449885 1178 log.cc:1198] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/49042c982df448e2b61ce38de50bf1a1
I20250110 02:51:09.449875 1214 leader_election.cc:290] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.450321 1178 ts_tablet_manager.cc:1939] T 49042c982df448e2b61ce38de50bf1a1 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.450455 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f307f4200409aaa47df55f20472d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.450673 1061 raft_consensus.cc:3054] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.451409 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 49042c982df448e2b61ce38de50bf1a1 (table default.Table_d5b4c03572aa4c8fae0339d6ea589418 [id=2ae8fc03d2a14acbbd1776f88d7fa63f]) successfully deleted
I20250110 02:51:09.449788 1188 tablet_bootstrap.cc:654] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.453478 1061 raft_consensus.cc:2463] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.453810 1187 raft_consensus.cc:491] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.454245 1187 raft_consensus.cc:513] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.455366 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe17b2fbe80d4468adf9adb20fd86676" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.455626 918 raft_consensus.cc:2463] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.456043 1188 tablet_bootstrap.cc:492] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.456012 709 leader_election.cc:304] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.456257 1188 ts_tablet_manager.cc:1397] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250110 02:51:09.456969 1188 raft_consensus.cc:357] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.457229 1188 raft_consensus.cc:383] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.457312 1187 leader_election.cc:290] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.457360 1188 raft_consensus.cc:738] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.457615 1214 raft_consensus.cc:2798] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.457648 1188 consensus_queue.cc:260] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.457811 1214 raft_consensus.cc:491] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.458004 1214 raft_consensus.cc:3054] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:09.458143 711 leader_election.cc:343] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:09.458223 1198 raft_consensus.cc:491] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.458334 1198 raft_consensus.cc:513] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.458923 1198 leader_election.cc:290] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.460392 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:09.460572 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.460700 776 raft_consensus.cc:2388] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:09.461632 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_61c719f45f3d4f1382d1fdb4954048d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.461838 1188 ts_tablet_manager.cc:1428] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:09.462357 1214 raft_consensus.cc:513] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.462906 1188 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:09.463042 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61c719f45f3d4f1382d1fdb4954048d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.463029 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.465044 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe17b2fbe80d4468adf9adb20fd86676" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.465315 918 raft_consensus.cc:3054] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.465883 1188 tablet_bootstrap.cc:654] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.466063 752 tablet_service.cc:1467] Processing CreateTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e (DEFAULT_TABLE table=default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.466547 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ab917bde534e3fbdf5dbae4af9304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.468492 918 raft_consensus.cc:2463] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.468945 709 leader_election.cc:304] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.469264 1187 raft_consensus.cc:2798] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.470521 1187 raft_consensus.cc:695] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.470947 1187 consensus_queue.cc:237] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.472781 1214 leader_election.cc:290] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:09.474153 711 leader_election.cc:343] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:09.474504 1188 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.474696 1188 ts_tablet_manager.cc:1397] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250110 02:51:09.474819 1179 tablet_replica.cc:331] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.475023 1179 raft_consensus.cc:2238] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:09.475276 1179 pending_rounds.cc:62] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:09.475399 1179 pending_rounds.cc:69] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611714957541376 op_type: NO_OP noop_request { }
I20250110 02:51:09.475581 1179 raft_consensus.cc:2883] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:09.475739 1179 raft_consensus.cc:2267] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.476313 1188 raft_consensus.cc:357] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.476464 1184 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.476583 1184 ts_tablet_manager.cc:1397] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.006s
I20250110 02:51:09.476596 1188 raft_consensus.cc:383] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.476696 1188 raft_consensus.cc:738] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.476703 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2da6483de7f5475ea3d755a6efbda369 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.477159 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6483de7f5475ea3d755a6efbda369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.477013 1188 consensus_queue.cc:260] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.477509 1184 raft_consensus.cc:357] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.477821 1188 ts_tablet_manager.cc:1428] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:09.478102 1188 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.477725 1184 raft_consensus.cc:383] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.479292 1184 raft_consensus.cc:738] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.479568 1184 consensus_queue.cc:260] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.481065 1184 ts_tablet_manager.cc:1428] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:09.481549 1184 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.481979 1179 ts_tablet_manager.cc:1905] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.483507 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c81e5a619cd643dea7b457f4916ecf68 (table default.Table_4fd336aafe184c11a74f324e2ac47f8d [id=fda4a3d90a014e28910b287af54f22c3]) successfully deleted
I20250110 02:51:09.483767 1184 tablet_bootstrap.cc:654] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.484303 1188 tablet_bootstrap.cc:654] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.484728 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 2da6483de7f5475ea3d755a6efbda369 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.485116 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6483de7f5475ea3d755a6efbda369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.488821 1179 ts_tablet_manager.cc:1918] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:09.488942 1179 log.cc:1198] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:09.489233 1179 ts_tablet_manager.cc:1939] T fe17b2fbe80d4468adf9adb20fd86676 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.489877 1179 tablet_replica.cc:331] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.490031 1179 raft_consensus.cc:2238] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:09.490217 1179 pending_rounds.cc:62] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:09.490271 1179 pending_rounds.cc:69] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611714508996608 op_type: NO_OP noop_request { }
I20250110 02:51:09.464808 1180 ts_tablet_manager.cc:1905] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.490408 1179 raft_consensus.cc:2883] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:09.490504 1179 raft_consensus.cc:2267] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.493235 1179 ts_tablet_manager.cc:1905] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.495873 1188 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.496099 1188 ts_tablet_manager.cc:1397] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20250110 02:51:09.496970 1188 raft_consensus.cc:357] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.497231 1188 raft_consensus.cc:383] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.497344 1188 raft_consensus.cc:738] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.497644 1188 consensus_queue.cc:260] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.498736 1188 ts_tablet_manager.cc:1428] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:09.499806 1188 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.501153 1179 ts_tablet_manager.cc:1918] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:09.501314 1179 log.cc:1198] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7059bcdf7e4d4cf4b559b00baf58f6c5
I20250110 02:51:09.501497 1188 tablet_bootstrap.cc:654] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.502588 1179 ts_tablet_manager.cc:1939] T 7059bcdf7e4d4cf4b559b00baf58f6c5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.503120 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet fe17b2fbe80d4468adf9adb20fd86676 (table default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]) successfully deleted
I20250110 02:51:09.503983 1179 tablet_replica.cc:331] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.504258 1179 raft_consensus.cc:2238] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.504513 1179 raft_consensus.cc:2267] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.505299 1179 ts_tablet_manager.cc:1905] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.508008 524 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.508896 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7059bcdf7e4d4cf4b559b00baf58f6c5 (table default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
W20250110 02:51:09.510602 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 866513b2738844bba19bb91a7f4bfd81 consensus state has no leader
I20250110 02:51:09.510962 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a9587fbd6e924c49b1a9a1189bff5d63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.511541 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9587fbd6e924c49b1a9a1189bff5d63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.514565 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 286aee6074e14c27b17bc79a78906620
I20250110 02:51:09.515007 518 catalog_manager.cc:6596] ReplaceTablet: tablet 286aee6074e14c27b17bc79a78906620 deleted and replaced by tablet fe86be26d1b544c08b231f20085d3159
I20250110 02:51:09.515224 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 286aee6074e14c27b17bc79a78906620 with delete_type TABLET_DATA_DELETED (replaced by tablet fe86be26d1b544c08b231f20085d3159) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.516227 518 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]). Expected version 1 got 0
I20250110 02:51:09.516501 518 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]). Expected version 1 got 0
I20250110 02:51:09.516587 1188 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.516607 1179 ts_tablet_manager.cc:1918] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.516716 1179 log.cc:1198] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/3b27ee8bd87f4316baccc8b85c84d2fd
I20250110 02:51:09.516731 1188 ts_tablet_manager.cc:1397] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20250110 02:51:09.517082 1179 ts_tablet_manager.cc:1939] T 3b27ee8bd87f4316baccc8b85c84d2fd P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.517725 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d4d87a97dad24a4784c3a82e4bf288f9" } new_table_name: "default.Table_3e5d4c967ed04133a245f4185b257710" modify_external_catalogs: true
I20250110 02:51:09.518025 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 3b27ee8bd87f4316baccc8b85c84d2fd (table default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]) successfully deleted
I20250110 02:51:09.518286 1179 tablet_replica.cc:331] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.518556 893 tablet_service.cc:1514] Processing DeleteTablet for tablet fe17b2fbe80d4468adf9adb20fd86676 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.518805 1179 raft_consensus.cc:2238] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.518946 1179 raft_consensus.cc:2267] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.519459 1178 tablet_replica.cc:331] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.519707 1178 raft_consensus.cc:2238] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.519636 1188 raft_consensus.cc:357] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.519856 1178 raft_consensus.cc:2267] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.519853 1188 raft_consensus.cc:383] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.520211 1188 raft_consensus.cc:738] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.520522 1178 ts_tablet_manager.cc:1905] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.520571 1188 consensus_queue.cc:260] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.521998 894 tablet_service.cc:1514] Processing DeleteTablet for tablet e91197c808ba4710970fcdb57f06d4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.522045 1188 ts_tablet_manager.cc:1428] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:09.522291 1188 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.523973 1188 tablet_bootstrap.cc:654] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.525260 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:09.526557 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.527490 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 286aee6074e14c27b17bc79a78906620 with delete_type TABLET_DATA_DELETED (replaced by tablet fe86be26d1b544c08b231f20085d3159) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.527501 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.528443 1178 ts_tablet_manager.cc:1918] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.528573 1178 log.cc:1198] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/fe17b2fbe80d4468adf9adb20fd86676
I20250110 02:51:09.528972 1178 ts_tablet_manager.cc:1939] T fe17b2fbe80d4468adf9adb20fd86676 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.528949 524 catalog_manager.cc:5526] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:09.530052 1178 tablet_replica.cc:331] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.530294 1178 raft_consensus.cc:2238] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.530486 1178 raft_consensus.cc:2267] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.530781 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 286aee6074e14c27b17bc79a78906620 with delete_type TABLET_DATA_DELETED (replaced by tablet fe86be26d1b544c08b231f20085d3159) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.530907 524 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]). Expected version 2 got 0
I20250110 02:51:09.531405 1179 ts_tablet_manager.cc:1905] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.532071 1178 ts_tablet_manager.cc:1905] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.533679 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b27ee8bd87f4316baccc8b85c84d2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:08 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.545562 1179 ts_tablet_manager.cc:1918] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.546669 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 89984fa8f39641a39969f4e3392ae8b7
I20250110 02:51:09.547372 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 2da6483de7f5475ea3d755a6efbda369 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.548697 1178 ts_tablet_manager.cc:1918] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.548899 1178 log.cc:1198] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e91197c808ba4710970fcdb57f06d4b9
I20250110 02:51:09.549304 1178 ts_tablet_manager.cc:1939] T e91197c808ba4710970fcdb57f06d4b9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.550412 1178 tablet_replica.cc:331] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.549806 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.550493 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.550806 1179 log.cc:1198] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b85eea2ad540445dbd95a655135dc043
I20250110 02:51:09.551259 1179 ts_tablet_manager.cc:1939] T b85eea2ad540445dbd95a655135dc043 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.552407 1179 tablet_replica.cc:331] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.552459 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b85eea2ad540445dbd95a655135dc043 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
I20250110 02:51:09.552805 1179 raft_consensus.cc:2238] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:09.553146 1179 pending_rounds.cc:62] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:09.553243 1179 pending_rounds.cc:69] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611714008727552 op_type: NO_OP noop_request { }
I20250110 02:51:09.553387 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 66f6f4f221e6445db0bb569f372cfaa0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.553436 1179 raft_consensus.cc:2883] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:09.553527 1179 raft_consensus.cc:2267] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.553788 1178 raft_consensus.cc:2238] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.553954 1178 raft_consensus.cc:2267] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:09.554634 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe86be26d1b544c08b231f20085d3159 has no consensus state
I20250110 02:51:09.555047 1179 ts_tablet_manager.cc:1905] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.555848 1178 ts_tablet_manager.cc:1905] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.556486 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b27ee8bd87f4316baccc8b85c84d2fd from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.557099 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_009422f290cd4fc9983b13a098c7b13b" } modify_external_catalogs: true
I20250110 02:51:09.557441 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_009422f290cd4fc9983b13a098c7b13b" } modify_external_catalogs: true
I20250110 02:51:09.562482 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.562765 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.563380 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet fe17b2fbe80d4468adf9adb20fd86676 (table default.Table_7d0a3876ce3d48d8b614244462399656 [id=6bd15ab926eb45e383ed46bf79311524]) successfully deleted
I20250110 02:51:09.563598 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e91197c808ba4710970fcdb57f06d4b9 (table default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]) successfully deleted
I20250110 02:51:09.570766 1178 ts_tablet_manager.cc:1918] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.572086 1179 ts_tablet_manager.cc:1918] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:09.572327 1179 log.cc:1198] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1c46be1162124b50ae43d8ea57cb5a0b
I20250110 02:51:09.572856 1179 ts_tablet_manager.cc:1939] T 1c46be1162124b50ae43d8ea57cb5a0b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.572989 1178 log.cc:1198] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/286aee6074e14c27b17bc79a78906620
I20250110 02:51:09.573122 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet a9f404a2e7494645af449aea3d3d3bf0
I20250110 02:51:09.573549 1178 ts_tablet_manager.cc:1939] T 286aee6074e14c27b17bc79a78906620 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.573660 521 catalog_manager.cc:6596] ReplaceTablet: tablet a9f404a2e7494645af449aea3d3d3bf0 deleted and replaced by tablet c65704c9d486444ebe8e4ffd4580f395
W20250110 02:51:09.573673 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c845c27a86d4e4f93c932313ebf648b consensus state has no leader
I20250110 02:51:09.574831 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.575160 1178 tablet_replica.cc:331] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.575909 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.575984 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]) successfully deleted
I20250110 02:51:09.576364 1179 tablet_replica.cc:331] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.576728 1179 raft_consensus.cc:2238] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.577032 1179 raft_consensus.cc:2267] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.575469 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.577519 1178 raft_consensus.cc:2238] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.577860 1178 raft_consensus.cc:2267] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.578438 1179 ts_tablet_manager.cc:1905] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.579036 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 286aee6074e14c27b17bc79a78906620 (table default.Table_ced8b781ad0f4d948f737825145e4dbf [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
I20250110 02:51:09.579418 1178 ts_tablet_manager.cc:1905] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.579809 1181 ts_tablet_manager.cc:1428] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.145s user 0.012s sys 0.026s
I20250110 02:51:09.580420 1181 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.580999 1187 raft_consensus.cc:491] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.581154 1187 raft_consensus.cc:513] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.581815 1187 leader_election.cc:290] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.582792 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c683bc3feaff44b1830cccfe839f677e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.582980 918 raft_consensus.cc:2463] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.583029 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c683bc3feaff44b1830cccfe839f677e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.583307 1060 raft_consensus.cc:2463] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.583434 709 leader_election.cc:304] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.583739 1187 raft_consensus.cc:2798] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.583849 1187 raft_consensus.cc:491] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.583986 1187 raft_consensus.cc:3054] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.584019 1181 tablet_bootstrap.cc:654] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.585784 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.585805 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24efc81a5ac743809665b17f941c0652" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.586004 918 raft_consensus.cc:2463] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:09.586616 1179 ts_tablet_manager.cc:1918] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.591003 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6365c4bf56b44696a42c21a809c8210c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.591576 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6365c4bf56b44696a42c21a809c8210c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.592406 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_eea9dbdd7e284980bec035a0b1dc0c4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.592924 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.593672 1178 ts_tablet_manager.cc:1918] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.586920 1187 raft_consensus.cc:513] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.594111 1181 tablet_bootstrap.cc:492] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.594275 1181 ts_tablet_manager.cc:1397] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20250110 02:51:09.594287 1187 leader_election.cc:290] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.595003 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c683bc3feaff44b1830cccfe839f677e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.595191 1060 raft_consensus.cc:3054] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.596743 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1eb291591c244ffa92efc9d213a760b9" } modify_external_catalogs: true
I20250110 02:51:09.596915 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1eb291591c244ffa92efc9d213a760b9" } modify_external_catalogs: true
I20250110 02:51:09.597652 1181 raft_consensus.cc:357] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.597856 1181 raft_consensus.cc:383] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.597956 1181 raft_consensus.cc:738] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.597959 1060 raft_consensus.cc:2463] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.598203 1181 consensus_queue.cc:260] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.598640 1179 log.cc:1198] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e91197c808ba4710970fcdb57f06d4b9
I20250110 02:51:09.598769 1181 ts_tablet_manager.cc:1428] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:09.598779 711 leader_election.cc:304] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.599092 1181 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.594978 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c683bc3feaff44b1830cccfe839f677e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.599220 1187 raft_consensus.cc:2798] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.599277 1178 log.cc:1198] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/3b27ee8bd87f4316baccc8b85c84d2fd
I20250110 02:51:09.599309 917 raft_consensus.cc:3054] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.599350 1187 raft_consensus.cc:695] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.599681 1178 ts_tablet_manager.cc:1939] T 3b27ee8bd87f4316baccc8b85c84d2fd P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.599579 1187 consensus_queue.cc:237] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.600656 1181 tablet_bootstrap.cc:654] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.601015 1179 ts_tablet_manager.cc:1939] T e91197c808ba4710970fcdb57f06d4b9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.601872 917 raft_consensus.cc:2463] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.601913 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e91197c808ba4710970fcdb57f06d4b9 (table default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]) successfully deleted
W20250110 02:51:09.602571 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 286aee6074e14c27b17bc79a78906620 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 286aee6074e14c27b17bc79a78906620 already in progress: creating tablet
I20250110 02:51:09.603096 1184 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.603250 1184 ts_tablet_manager.cc:1397] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.122s user 0.014s sys 0.015s
W20250110 02:51:09.604204 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2a27f0327d1844b8a561af8b0bf4385b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a27f0327d1844b8a561af8b0bf4385b already in progress: creating tablet
I20250110 02:51:09.604317 1184 raft_consensus.cc:357] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.604552 1184 raft_consensus.cc:383] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.604498 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 3b27ee8bd87f4316baccc8b85c84d2fd (table default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]) successfully deleted
I20250110 02:51:09.604635 1184 raft_consensus.cc:738] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:09.604856 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2a27f0327d1844b8a561af8b0bf4385b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a27f0327d1844b8a561af8b0bf4385b already in progress: creating tablet
I20250110 02:51:09.604880 1184 consensus_queue.cc:260] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:09.605157 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 89984fa8f39641a39969f4e3392ae8b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89984fa8f39641a39969f4e3392ae8b7 already in progress: creating tablet
I20250110 02:51:09.605626 1184 ts_tablet_manager.cc:1428] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:09.605973 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_78653a7a5f0743f18b98b3e703c05a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.606395 1184 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:09.606504 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78653a7a5f0743f18b98b3e703c05a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:09.606953 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a9f404a2e7494645af449aea3d3d3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f404a2e7494645af449aea3d3d3bf0 already in progress: creating tablet
I20250110 02:51:09.608161 1184 tablet_bootstrap.cc:654] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:09.609869 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 89984fa8f39641a39969f4e3392ae8b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89984fa8f39641a39969f4e3392ae8b7 already in progress: creating tablet
W20250110 02:51:09.610225 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a9f404a2e7494645af449aea3d3d3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f404a2e7494645af449aea3d3d3bf0 already in progress: creating tablet
I20250110 02:51:09.613243 1180 ts_tablet_manager.cc:1918] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.613389 1180 log.cc:1198] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e91197c808ba4710970fcdb57f06d4b9
I20250110 02:51:09.613744 1180 ts_tablet_manager.cc:1939] T e91197c808ba4710970fcdb57f06d4b9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.615384 1184 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.615924 1184 ts_tablet_manager.cc:1397] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250110 02:51:09.616918 1181 tablet_bootstrap.cc:492] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.617997 1184 raft_consensus.cc:357] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.618479 1181 ts_tablet_manager.cc:1397] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250110 02:51:09.619526 1181 raft_consensus.cc:357] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.619871 1184 raft_consensus.cc:383] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.620034 1184 raft_consensus.cc:738] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.620380 1184 consensus_queue.cc:260] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.620702 1181 raft_consensus.cc:383] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.621289 1184 ts_tablet_manager.cc:1428] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:09.621425 1181 raft_consensus.cc:738] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.621690 1181 consensus_queue.cc:260] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.623466 1184 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.623785 1181 ts_tablet_manager.cc:1428] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:09.625857 1184 tablet_bootstrap.cc:654] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.626255 1181 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.626170 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_bcc7554985664d30b2fca81e32af5394"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.626922 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcc7554985664d30b2fca81e32af5394 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.628475 1181 tablet_bootstrap.cc:654] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.640305 1181 tablet_bootstrap.cc:492] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.640472 1181 ts_tablet_manager.cc:1397] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20250110 02:51:09.641098 1181 raft_consensus.cc:357] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.641322 1181 raft_consensus.cc:383] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.641408 1181 raft_consensus.cc:738] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.642081 1181 consensus_queue.cc:260] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.642421 1183 raft_consensus.cc:695] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:09.642618 1181 ts_tablet_manager.cc:1428] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:09.642647 1184 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.642866 1184 ts_tablet_manager.cc:1397] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20250110 02:51:09.642912 1181 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.643725 1184 raft_consensus.cc:357] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.644011 1184 raft_consensus.cc:383] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.644109 1184 raft_consensus.cc:738] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.643867 1183 consensus_queue.cc:237] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.644526 1181 tablet_bootstrap.cc:654] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.644444 1184 consensus_queue.cc:260] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.646560 1258 raft_consensus.cc:491] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.646709 1258 raft_consensus.cc:513] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.647423 1258 leader_election.cc:290] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:09.648300 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286aee6074e14c27b17bc79a78906620" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:09.648491 776 raft_consensus.cc:2463] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:09.649024 1184 ts_tablet_manager.cc:1428] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250110 02:51:09.649360 996 leader_election.cc:304] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.649819 1258 raft_consensus.cc:2798] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.649924 1258 raft_consensus.cc:491] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.649993 1258 raft_consensus.cc:3054] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:09.650689 994 leader_election.cc:343] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 286aee6074e14c27b17bc79a78906620
I20250110 02:51:09.651185 1184 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.652894 1184 tablet_bootstrap.cc:654] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.653350 1258 raft_consensus.cc:513] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.654390 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286aee6074e14c27b17bc79a78906620" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:09.654608 776 raft_consensus.cc:3054] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.657281 776 raft_consensus.cc:2463] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:09.657773 1184 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.657934 1184 ts_tablet_manager.cc:1397] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20250110 02:51:09.658007 996 leader_election.cc:304] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.658922 1184 raft_consensus.cc:357] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.659162 1184 raft_consensus.cc:383] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.659276 1184 raft_consensus.cc:738] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:09.659518 994 leader_election.cc:343] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 286aee6074e14c27b17bc79a78906620
I20250110 02:51:09.659304 1258 leader_election.cc:290] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:09.659590 1184 consensus_queue.cc:260] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.660058 1258 raft_consensus.cc:2798] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.660209 1258 raft_consensus.cc:695] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:09.660485 1258 consensus_queue.cc:237] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.663574 853 leader_election.cc:304] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:09.664084 1198 raft_consensus.cc:3054] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.665004 1181 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.665151 1181 ts_tablet_manager.cc:1397] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250110 02:51:09.666128 1181 raft_consensus.cc:357] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.666357 1181 raft_consensus.cc:383] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.666481 1181 raft_consensus.cc:738] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.666728 1181 consensus_queue.cc:260] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.669939 1198 raft_consensus.cc:2743] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:09.671406 1184 ts_tablet_manager.cc:1428] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250110 02:51:09.675868 1184 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.678200 1184 tablet_bootstrap.cc:654] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:09.681442 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05d5f9bda55942ff856252bbb50b3c5b consensus state has no leader
I20250110 02:51:09.684767 1184 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.684962 1184 ts_tablet_manager.cc:1397] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20250110 02:51:09.685757 1184 raft_consensus.cc:357] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.685958 1184 raft_consensus.cc:383] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.686022 1184 raft_consensus.cc:738] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.686273 1184 consensus_queue.cc:260] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.689575 1184 ts_tablet_manager.cc:1428] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:09.689882 1184 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.691473 1184 tablet_bootstrap.cc:654] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.701190 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4f862664cf4044afbe096ffdf7960b32" } new_table_name: "default.Table_1a58fb7be3a34571bedcfbbff5db8ea7" modify_external_catalogs: true
I20250110 02:51:09.715674 1181 ts_tablet_manager.cc:1428] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.050s user 0.022s sys 0.014s
I20250110 02:51:09.715862 1184 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.716007 1184 ts_tablet_manager.cc:1397] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.003s
I20250110 02:51:09.716115 1181 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.718883 1181 tablet_bootstrap.cc:654] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.721753 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:09.722270 522 catalog_manager.cc:6596] ReplaceTablet: tablet 2da6483de7f5475ea3d755a6efbda369 deleted and replaced by tablet ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:09.722615 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:09.723232 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2da6483de7f5475ea3d755a6efbda369 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2da6483de7f5475ea3d755a6efbda369 already in progress: creating tablet
I20250110 02:51:09.723349 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.723677 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:09.724131 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2da6483de7f5475ea3d755a6efbda369 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2da6483de7f5475ea3d755a6efbda369 already in progress: creating tablet
I20250110 02:51:09.724805 1184 raft_consensus.cc:357] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.725056 1184 raft_consensus.cc:383] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.725142 1184 raft_consensus.cc:738] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.725390 1184 consensus_queue.cc:260] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.725953 1184 ts_tablet_manager.cc:1428] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:09.726210 1184 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.728183 1184 tablet_bootstrap.cc:654] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.729071 1180 tablet_replica.cc:331] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:09.729146 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e91197c808ba4710970fcdb57f06d4b9 (table default.Table_b5cee20c2ba547278df5e4649d16f0b0 [id=a0bf06a2dbae49acb26a788f2da09b97]) successfully deleted
I20250110 02:51:09.729274 1180 raft_consensus.cc:2238] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.729429 1180 raft_consensus.cc:2267] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.730135 1180 ts_tablet_manager.cc:1905] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.730732 1181 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.730932 1181 ts_tablet_manager.cc:1397] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.002s
I20250110 02:51:09.731679 1181 raft_consensus.cc:357] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.731904 1181 raft_consensus.cc:383] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.731973 1188 tablet_bootstrap.cc:492] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.731982 1181 raft_consensus.cc:738] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.732128 1188 ts_tablet_manager.cc:1397] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.210s user 0.057s sys 0.104s
I20250110 02:51:09.732245 1181 consensus_queue.cc:260] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.732708 1181 ts_tablet_manager.cc:1428] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:09.732800 1188 raft_consensus.cc:357] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.732983 1181 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.733035 1188 raft_consensus.cc:383] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.733112 1188 raft_consensus.cc:738] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.733331 1188 consensus_queue.cc:260] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.735062 1188 ts_tablet_manager.cc:1428] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:09.735329 1188 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.735796 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 66f6f4f221e6445db0bb569f372cfaa0
I20250110 02:51:09.736258 517 catalog_manager.cc:6596] ReplaceTablet: tablet 66f6f4f221e6445db0bb569f372cfaa0 deleted and replaced by tablet 825b4d370dab4b908ddde0f1dddf466f
I20250110 02:51:09.737023 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.737140 1188 tablet_bootstrap.cc:654] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:09.737720 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 66f6f4f221e6445db0bb569f372cfaa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66f6f4f221e6445db0bb569f372cfaa0 already in progress: creating tablet
I20250110 02:51:09.737931 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_4813a1a74b1a455584f21646ec5a97d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.738996 1181 tablet_bootstrap.cc:654] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.742449 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.742710 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.743175 1178 tablet_replica.cc:331] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.743422 1178 raft_consensus.cc:2238] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.743610 1178 raft_consensus.cc:2267] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.744360 1178 ts_tablet_manager.cc:1905] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:09.744550 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 66f6f4f221e6445db0bb569f372cfaa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66f6f4f221e6445db0bb569f372cfaa0 already in progress: creating tablet
I20250110 02:51:09.744947 1188 tablet_bootstrap.cc:492] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.745077 1188 ts_tablet_manager.cc:1397] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20250110 02:51:09.746279 1188 raft_consensus.cc:357] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.746500 1188 raft_consensus.cc:383] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.746591 1188 raft_consensus.cc:738] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.746798 1188 consensus_queue.cc:260] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.749230 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.749702 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.754144 1188 ts_tablet_manager.cc:1428] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250110 02:51:09.754123 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7192835167d842589c56b4b5432a247f" } modify_external_catalogs: true
I20250110 02:51:09.754302 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7192835167d842589c56b4b5432a247f" } modify_external_catalogs: true
I20250110 02:51:09.754590 1188 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.754985 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_61c719f45f3d4f1382d1fdb4954048d6" } new_table_name: "default.Table_7c26cb6440db4eb8a73ac05566a26a75" modify_external_catalogs: true
I20250110 02:51:09.756358 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ff78b6f4c98a42b69c23212d1fa6f09b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.756810 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff78b6f4c98a42b69c23212d1fa6f09b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.756969 1178 ts_tablet_manager.cc:1918] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.757076 1178 log.cc:1198] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:09.757315 1188 tablet_bootstrap.cc:654] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.757462 1178 ts_tablet_manager.cc:1939] T b851ddf27fe44d19b9ca39d82e6352cd P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:09.762557 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b851ddf27fe44d19b9ca39d82e6352cd (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
I20250110 02:51:09.763089 1184 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.763278 1184 ts_tablet_manager.cc:1397] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250110 02:51:09.764178 1184 raft_consensus.cc:357] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.764443 1184 raft_consensus.cc:383] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.764518 1184 raft_consensus.cc:738] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.764838 1184 consensus_queue.cc:260] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.768879 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.769475 1184 ts_tablet_manager.cc:1428] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250110 02:51:09.769801 1184 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.772677 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 886ab6d12cf545b9b700959e18396c50 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.773699 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b85eea2ad540445dbd95a655135dc043 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da6483de7f5475ea3d755a6efbda369) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.774250 1187 raft_consensus.cc:491] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.774428 1187 raft_consensus.cc:513] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.775096 1187 leader_election.cc:290] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.775238 1178 tablet_replica.cc:331] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:09.775491 1178 raft_consensus.cc:2238] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.775678 1178 raft_consensus.cc:2267] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.776105 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58e92b9789e49f6bbfbefe3f8f3a0ef" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.776417 917 raft_consensus.cc:2463] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.776831 709 leader_election.cc:304] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.777488 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58e92b9789e49f6bbfbefe3f8f3a0ef" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.777738 1061 raft_consensus.cc:2463] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:09.778611 1187 raft_consensus.cc:2798] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.778728 1187 raft_consensus.cc:491] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.778811 1187 raft_consensus.cc:3054] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.781332 1187 raft_consensus.cc:513] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.782223 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58e92b9789e49f6bbfbefe3f8f3a0ef" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.782431 917 raft_consensus.cc:3054] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.784845 917 raft_consensus.cc:2463] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.785250 709 leader_election.cc:304] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:09.786123 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58e92b9789e49f6bbfbefe3f8f3a0ef" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.786342 1061 raft_consensus.cc:3054] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:09.786783 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4813a1a74b1a455584f21646ec5a97d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.787045 1214 raft_consensus.cc:2798] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.787179 1214 raft_consensus.cc:695] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.787393 1214 consensus_queue.cc:237] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.788798 1061 raft_consensus.cc:2463] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.789718 518 catalog_manager.cc:5526] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:09.790288 1187 leader_election.cc:290] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.791965 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 10504e77a5c64ebf85be937d64ca9c80 (table default.Table_4f862664cf4044afbe096ffdf7960b32 [id=dd4008ea14ad40c68b3e71fac9e2be88]). Expected version 1 got 0
I20250110 02:51:09.792179 522 catalog_manager.cc:5526] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:09.792479 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]). Expected version 1 got 0
I20250110 02:51:09.792698 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]). Expected version 1 got 0
I20250110 02:51:09.792891 1181 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.793038 1181 ts_tablet_manager.cc:1397] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.002s
W20250110 02:51:09.793406 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10504e77a5c64ebf85be937d64ca9c80 consensus state has no leader
I20250110 02:51:09.793926 1181 raft_consensus.cc:357] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.794144 1181 raft_consensus.cc:383] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.794234 1181 raft_consensus.cc:738] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.794535 1181 consensus_queue.cc:260] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.778625 1184 tablet_bootstrap.cc:654] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.796911 1181 ts_tablet_manager.cc:1428] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:09.798475 1187 raft_consensus.cc:491] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.798615 1187 raft_consensus.cc:513] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.799293 1187 leader_election.cc:290] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.799441 1181 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.799950 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b851ddf27fe44d19b9ca39d82e6352cd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.800155 1061 raft_consensus.cc:2463] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:09.800331 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2c01af512f460daa98b19444f72172 has no consensus state
I20250110 02:51:09.800657 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a1233829553f48db897d4e93aa37c38d" } modify_external_catalogs: true
W20250110 02:51:09.800771 709 leader_election.cc:343] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:09.800838 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a1233829553f48db897d4e93aa37c38d" } modify_external_catalogs: true
I20250110 02:51:09.801124 711 leader_election.cc:304] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.801848 1187 raft_consensus.cc:2798] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.801956 1187 raft_consensus.cc:491] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.801898 895 tablet_service.cc:1467] Processing CreateTablet for tablet aa40d20288644b2a8d36ede967d16e79 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.802031 1187 raft_consensus.cc:3054] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.802259 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa40d20288644b2a8d36ede967d16e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.802834 748 tablet_service.cc:1467] Processing CreateTablet for tablet d98b789eaabe4191af40c9f713acb030 (DEFAULT_TABLE table=default.Table_5f6d11dc852b4cd189790829bca600ce [id=f3234e13443740dda53c51aecc6cd8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.803169 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98b789eaabe4191af40c9f713acb030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.804517 1187 raft_consensus.cc:513] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.805403 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2412420dc54258a6921ead194c1587 (DEFAULT_TABLE table=default.Table_b691c5c8267948a08b5a76dc5675aa52 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.805778 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2412420dc54258a6921ead194c1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.805938 756 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3f60c819f4824bc99d755e7442976 (DEFAULT_TABLE table=default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.806293 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3f60c819f4824bc99d755e7442976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.806659 893 tablet_service.cc:1467] Processing CreateTablet for tablet d98b789eaabe4191af40c9f713acb030 (DEFAULT_TABLE table=default.Table_5f6d11dc852b4cd189790829bca600ce [id=f3234e13443740dda53c51aecc6cd8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.807013 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98b789eaabe4191af40c9f713acb030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.807464 1040 tablet_service.cc:1467] Processing CreateTablet for tablet d98b789eaabe4191af40c9f713acb030 (DEFAULT_TABLE table=default.Table_5f6d11dc852b4cd189790829bca600ce [id=f3234e13443740dda53c51aecc6cd8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.807796 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98b789eaabe4191af40c9f713acb030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.809640 894 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2412420dc54258a6921ead194c1587 (DEFAULT_TABLE table=default.Table_b691c5c8267948a08b5a76dc5675aa52 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.809731 891 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3f60c819f4824bc99d755e7442976 (DEFAULT_TABLE table=default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:09.809923 709 leader_election.cc:343] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:09.810001 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2412420dc54258a6921ead194c1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.810225 752 tablet_service.cc:1467] Processing CreateTablet for tablet aa40d20288644b2a8d36ede967d16e79 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.810693 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2412420dc54258a6921ead194c1587 (DEFAULT_TABLE table=default.Table_b691c5c8267948a08b5a76dc5675aa52 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.811018 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2412420dc54258a6921ead194c1587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.811512 1187 leader_election.cc:290] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.813678 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa40d20288644b2a8d36ede967d16e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.814507 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3f60c819f4824bc99d755e7442976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.814626 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3f60c819f4824bc99d755e7442976 (DEFAULT_TABLE table=default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.819275 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3f60c819f4824bc99d755e7442976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.824179 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b851ddf27fe44d19b9ca39d82e6352cd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.824369 1061 raft_consensus.cc:3054] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.824753 1041 tablet_service.cc:1467] Processing CreateTablet for tablet aa40d20288644b2a8d36ede967d16e79 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.825232 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa40d20288644b2a8d36ede967d16e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.828905 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1aafff2e0de548129d99e9f07dc3ab11
I20250110 02:51:09.829084 1061 raft_consensus.cc:2463] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.829639 711 leader_election.cc:304] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.829934 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5f6d11dc852b4cd189790829bca600ce" } new_table_name: "default.Table_8c550622172349e2afb560f02401b8fa" modify_external_catalogs: true
I20250110 02:51:09.830211 1187 raft_consensus.cc:2798] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.830375 1187 raft_consensus.cc:695] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.830660 1187 consensus_queue.cc:237] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.832201 1181 tablet_bootstrap.cc:654] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.833684 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c9d791b1c504171ba66dec55c8fb0e6 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.835105 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.835335 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_046451c051e2479b8b7de432ffb408bc" } modify_external_catalogs: true
I20250110 02:51:09.835479 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_046451c051e2479b8b7de432ffb408bc" } modify_external_catalogs: true
W20250110 02:51:09.835882 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1aafff2e0de548129d99e9f07dc3ab11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aafff2e0de548129d99e9f07dc3ab11 already in progress: creating tablet
I20250110 02:51:09.837308 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.837422 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ced8b781ad0f4d948f737825145e4dbf" } new_table_name: "default.Table_b439c40f07fd4b48b1735826b7eedbf0" modify_external_catalogs: true
I20250110 02:51:09.840176 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.851948 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 02d3f60c819f4824bc99d755e7442976
I20250110 02:51:09.852475 525 master_service.cc:946] ReplaceTablet: received request to replace tablet a9f404a2e7494645af449aea3d3d3bf0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.852748 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.853046 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a037268bb40540c7b7495367b363d30b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.853581 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a037268bb40540c7b7495367b363d30b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:09.854663 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 02d3f60c819f4824bc99d755e7442976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d3f60c819f4824bc99d755e7442976 already in progress: creating tablet
I20250110 02:51:09.855109 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.855401 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.856912 1181 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.857076 1181 ts_tablet_manager.cc:1397] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.003s
I20250110 02:51:09.857192 895 tablet_service.cc:1467] Processing CreateTablet for tablet 8162ceb17c714614b34beb33a7633ff0 (DEFAULT_TABLE table=default.Table_627ff2f24b97485a9f948b72f8b39daf [id=3e8497573a584513ab1df4ba5ed27993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.857699 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8162ceb17c714614b34beb33a7633ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.857959 1181 raft_consensus.cc:357] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.858067 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8162ceb17c714614b34beb33a7633ff0 (DEFAULT_TABLE table=default.Table_627ff2f24b97485a9f948b72f8b39daf [id=3e8497573a584513ab1df4ba5ed27993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.858151 1181 raft_consensus.cc:383] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.858219 1181 raft_consensus.cc:738] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.858548 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8162ceb17c714614b34beb33a7633ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.858572 1181 consensus_queue.cc:260] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.859078 1181 ts_tablet_manager.cc:1428] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:09.859423 1181 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.861179 1181 tablet_bootstrap.cc:654] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.870491 1214 raft_consensus.cc:491] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.870652 1214 raft_consensus.cc:513] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.871270 1214 leader_election.cc:290] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.872123 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89984fa8f39641a39969f4e3392ae8b7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:09.872180 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89984fa8f39641a39969f4e3392ae8b7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.872370 1061 raft_consensus.cc:2463] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:09.872782 709 leader_election.cc:343] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.873191 711 leader_election.cc:304] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.873221 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 8162ceb17c714614b34beb33a7633ff0 (DEFAULT_TABLE table=default.Table_627ff2f24b97485a9f948b72f8b39daf [id=3e8497573a584513ab1df4ba5ed27993]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:09.873543 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8162ceb17c714614b34beb33a7633ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:09.873541 1214 raft_consensus.cc:2798] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.873631 1214 raft_consensus.cc:491] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.873696 1214 raft_consensus.cc:3054] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.876799 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 10ab917bde534e3fbdf5dbae4af9304e
I20250110 02:51:09.877177 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 886ab6d12cf545b9b700959e18396c50
I20250110 02:51:09.877626 526 catalog_manager.cc:6596] ReplaceTablet: tablet 886ab6d12cf545b9b700959e18396c50 deleted and replaced by tablet bd6d4c4631994a5792200ba4d4c906f0
I20250110 02:51:09.877527 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0595ee2f9d4442cca8e712debfae068e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.878049 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0595ee2f9d4442cca8e712debfae068e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.877992 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_033d0d58d92e4a23ae7b1927e5db6163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:09.880190 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.880384 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 886ab6d12cf545b9b700959e18396c50 with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d4c4631994a5792200ba4d4c906f0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.880450 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 886ab6d12cf545b9b700959e18396c50 with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d4c4631994a5792200ba4d4c906f0) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.880285 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.881109 1214 raft_consensus.cc:513] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.882145 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89984fa8f39641a39969f4e3392ae8b7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:09.882584 1179 tablet_replica.cc:331] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:09.882812 709 leader_election.cc:343] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.882836 1179 raft_consensus.cc:2238] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.883118 1179 raft_consensus.cc:2267] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:09.883375 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 10ab917bde534e3fbdf5dbae4af9304e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10ab917bde534e3fbdf5dbae4af9304e already in progress: creating tablet
I20250110 02:51:09.883756 1179 ts_tablet_manager.cc:1905] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:09.884261 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_033d0d58d92e4a23ae7b1927e5db6163 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:09.886662 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d98b789eaabe4191af40c9f713acb030 consensus state has no leader
I20250110 02:51:09.888967 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89984fa8f39641a39969f4e3392ae8b7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.889164 1061 raft_consensus.cc:3054] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.891538 1061 raft_consensus.cc:2463] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:09.891750 1279 raft_consensus.cc:491] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.891867 1279 raft_consensus.cc:513] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.892097 711 leader_election.cc:304] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.892359 1187 raft_consensus.cc:2798] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.892632 1279 leader_election.cc:290] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:09.892665 1187 raft_consensus.cc:695] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:09.892925 1187 consensus_queue.cc:237] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.893210 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:09.893844 994 leader_election.cc:343] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.895187 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:09.895377 776 raft_consensus.cc:2463] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:09.895725 1181 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.895799 996 leader_election.cc:304] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.896206 1279 raft_consensus.cc:2798] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.896685 1279 raft_consensus.cc:491] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.896811 1279 raft_consensus.cc:3054] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.896924 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 886ab6d12cf545b9b700959e18396c50 with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d4c4631994a5792200ba4d4c906f0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.883177 1214 leader_election.cc:290] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.900131 1279 raft_consensus.cc:513] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.900228 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b164fa0aebbe4bb2821b21038f738d0c" } new_table_name: "default.Table_8eb6afa433004d3aaac88418b07a91f3" modify_external_catalogs: true
I20250110 02:51:09.900939 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.900274 1179 ts_tablet_manager.cc:1918] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.901258 1179 log.cc:1198] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/886ab6d12cf545b9b700959e18396c50
I20250110 02:51:09.901679 1179 ts_tablet_manager.cc:1939] T 886ab6d12cf545b9b700959e18396c50 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.901584 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:09.902112 994 leader_election.cc:343] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:09.902496 1279 leader_election.cc:290] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:09.902560 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:09.902757 776 raft_consensus.cc:3054] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.902866 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 886ab6d12cf545b9b700959e18396c50 (table default.Table_b164fa0aebbe4bb2821b21038f738d0c [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]) successfully deleted
I20250110 02:51:09.905658 776 raft_consensus.cc:2463] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:09.905949 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 24efc81a5ac743809665b17f941c0652
I20250110 02:51:09.906126 996 leader_election.cc:304] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:09.895898 1181 ts_tablet_manager.cc:1397] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.007s
I20250110 02:51:09.906842 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 24efc81a5ac743809665b17f941c0652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.907176 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 24efc81a5ac743809665b17f941c0652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.907204 1279 raft_consensus.cc:2798] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.907522 1279 raft_consensus.cc:695] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:09.908064 1279 consensus_queue.cc:237] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:09.910128 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b85eea2ad540445dbd95a655135dc043 with delete_type TABLET_DATA_DELETED (replaced by tablet 2da6483de7f5475ea3d755a6efbda369) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.911480 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 24efc81a5ac743809665b17f941c0652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.911532 1179 tablet_replica.cc:331] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:09.911774 1179 raft_consensus.cc:2238] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:09.911948 1179 raft_consensus.cc:2267] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:09.912469 1179 ts_tablet_manager.cc:1905] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.914273 1198 raft_consensus.cc:491] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:09.914507 1198 raft_consensus.cc:513] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.915225 1198 leader_election.cc:290] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.916333 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7422cfed02fc4b50ab867fdf34769b58" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:09.916571 1061 raft_consensus.cc:2463] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:09.917069 853 leader_election.cc:304] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:09.917416 1198 raft_consensus.cc:2798] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:09.917541 1198 raft_consensus.cc:491] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:09.917641 1198 raft_consensus.cc:3054] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.776432 1178 ts_tablet_manager.cc:1905] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:09.921098 1198 raft_consensus.cc:513] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.921820 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7422cfed02fc4b50ab867fdf34769b58" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:09.922077 776 raft_consensus.cc:2463] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:09.922355 1198 leader_election.cc:290] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:09.922926 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7422cfed02fc4b50ab867fdf34769b58" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:09.923152 1060 raft_consensus.cc:3054] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.925876 1060 raft_consensus.cc:2463] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:09.926951 853 leader_election.cc:304] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:09.927490 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7422cfed02fc4b50ab867fdf34769b58" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:09.927722 776 raft_consensus.cc:3054] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:09.929461 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 7422cfed02fc4b50ab867fdf34769b58 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:09.930965 1198 raft_consensus.cc:2798] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:09.932785 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 4c9d791b1c504171ba66dec55c8fb0e6
I20250110 02:51:09.933854 776 raft_consensus.cc:2463] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:09.933976 524 catalog_manager.cc:6596] ReplaceTablet: tablet 4c9d791b1c504171ba66dec55c8fb0e6 deleted and replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:09.935292 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:09.935447 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:09.936753 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:09.937750 1184 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.937906 1184 ts_tablet_manager.cc:1397] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.168s user 0.009s sys 0.021s
I20250110 02:51:09.938761 1184 raft_consensus.cc:357] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.938982 1184 raft_consensus.cc:383] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.939078 1184 raft_consensus.cc:738] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.939333 1184 consensus_queue.cc:260] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.939882 1184 ts_tablet_manager.cc:1428] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:09.940199 1184 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.942940 1198 raft_consensus.cc:695] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:09.944586 1184 tablet_bootstrap.cc:654] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.944859 1198 consensus_queue.cc:237] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.907415 1181 raft_consensus.cc:357] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.949260 1181 raft_consensus.cc:383] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.949436 1181 raft_consensus.cc:738] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.949791 1181 consensus_queue.cc:260] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.951802 1181 ts_tablet_manager.cc:1428] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20250110 02:51:09.952062 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a788c9c301f14b34b873006c6715a243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.952648 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a788c9c301f14b34b873006c6715a243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.952253 1181 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.954913 1188 tablet_bootstrap.cc:492] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.955089 1188 ts_tablet_manager.cc:1397] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.201s user 0.026s sys 0.053s
I20250110 02:51:09.955914 1188 raft_consensus.cc:357] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.956115 1188 raft_consensus.cc:383] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.956202 1188 raft_consensus.cc:738] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.956584 1188 consensus_queue.cc:260] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.957159 1188 ts_tablet_manager.cc:1428] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:09.959975 1181 tablet_bootstrap.cc:654] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.960233 1184 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:09.960418 1184 ts_tablet_manager.cc:1397] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250110 02:51:09.961249 1184 raft_consensus.cc:357] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.961500 1184 raft_consensus.cc:383] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.961581 1184 raft_consensus.cc:738] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.961834 1184 consensus_queue.cc:260] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.962251 1179 ts_tablet_manager.cc:1918] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.962878 1188 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.964215 1180 ts_tablet_manager.cc:1918] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:09.965492 1188 tablet_bootstrap.cc:654] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.972548 917 raft_consensus.cc:1270] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Refusing update from remote peer 98336b684b41407da8a1949e30cd82e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:09.968230 1061 raft_consensus.cc:1270] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Refusing update from remote peer 98336b684b41407da8a1949e30cd82e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:09.974315 1180 log.cc:1198] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/3b27ee8bd87f4316baccc8b85c84d2fd
I20250110 02:51:09.974740 1180 ts_tablet_manager.cc:1939] T 3b27ee8bd87f4316baccc8b85c84d2fd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:09.975745 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 3b27ee8bd87f4316baccc8b85c84d2fd (table default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034]) successfully deleted
I20250110 02:51:09.979008 1181 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:09.979163 1181 ts_tablet_manager.cc:1397] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
I20250110 02:51:09.979841 1181 raft_consensus.cc:357] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.980074 1181 raft_consensus.cc:383] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.980156 1181 raft_consensus.cc:738] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.980396 1181 consensus_queue.cc:260] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:09.980854 1181 ts_tablet_manager.cc:1428] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:09.981163 1181 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:09.983008 1188 tablet_bootstrap.cc:492] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:09.962343 1184 ts_tablet_manager.cc:1428] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:09.983165 1188 ts_tablet_manager.cc:1397] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.012s
I20250110 02:51:09.973896 1187 consensus_queue.cc:1035] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:09.983482 1184 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:09.983610 1179 log.cc:1198] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/24efc81a5ac743809665b17f941c0652
I20250110 02:51:09.984053 1179 ts_tablet_manager.cc:1939] T 24efc81a5ac743809665b17f941c0652 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:09.984136 1188 raft_consensus.cc:357] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.984334 1188 raft_consensus.cc:383] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:09.984416 1188 raft_consensus.cc:738] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:09.984629 1188 consensus_queue.cc:260] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:09.984798 1214 consensus_queue.cc:1035] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:09.985147 1188 ts_tablet_manager.cc:1428] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:09.985168 1184 tablet_bootstrap.cc:654] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.990486 1181 tablet_bootstrap.cc:654] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:09.994202 1188 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:09.994463 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_335d6489a4484f1fb0de9408d54cfab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:09.995043 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_335d6489a4484f1fb0de9408d54cfab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:09.998143 1180 tablet_replica.cc:331] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.000164 1180 raft_consensus.cc:2238] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.000473 1180 raft_consensus.cc:2267] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.002629 1180 ts_tablet_manager.cc:1905] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.002655 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 24efc81a5ac743809665b17f941c0652 (table default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]) successfully deleted
I20250110 02:51:10.003682 1188 tablet_bootstrap.cc:654] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:10.004673 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4c9d791b1c504171ba66dec55c8fb0e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9d791b1c504171ba66dec55c8fb0e6 already in progress: creating tablet
I20250110 02:51:10.011569 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1621908f975348db834b77c8e899eb67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.012266 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1621908f975348db834b77c8e899eb67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.013237 1293 mvcc.cc:204] Tried to move back new op lower bound from 7112611716893949952 to 7112611714836803584. Current Snapshot: MvccSnapshot[applied={T|T < 7112611716893949952}]
I20250110 02:51:10.004463 1279 raft_consensus.cc:491] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.019089 1279 raft_consensus.cc:513] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.019948 1279 leader_election.cc:290] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.021546 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b85eea2ad540445dbd95a655135dc043" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:10.022358 994 leader_election.cc:343] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: Tablet not RUNNING: STOPPED
W20250110 02:51:10.022846 996 leader_election.cc:343] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b85eea2ad540445dbd95a655135dc043
I20250110 02:51:10.022986 996 leader_election.cc:304] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.023525 1279 raft_consensus.cc:2743] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.026921 1279 raft_consensus.cc:491] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.027097 1279 raft_consensus.cc:513] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.027796 1279 leader_election.cc:290] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.028544 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10504e77a5c64ebf85be937d64ca9c80" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:10.028831 776 raft_consensus.cc:2463] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:10.029707 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10504e77a5c64ebf85be937d64ca9c80" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.029973 918 raft_consensus.cc:2463] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:10.030649 996 leader_election.cc:304] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:10.031281 1279 raft_consensus.cc:2798] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:10.031380 1279 raft_consensus.cc:491] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:10.031477 1279 raft_consensus.cc:3054] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:10.032263 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd6d4c4631994a5792200ba4d4c906f0 has no consensus state
I20250110 02:51:10.032567 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 9bce8f5888bb404696e46487176b5fe8 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.034791 1279 raft_consensus.cc:513] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.036655 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10504e77a5c64ebf85be937d64ca9c80" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:10.036965 776 raft_consensus.cc:3054] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.039220 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10504e77a5c64ebf85be937d64ca9c80" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:10.039459 918 raft_consensus.cc:3054] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.041359 1178 ts_tablet_manager.cc:1918] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.041676 1178 log.cc:1198] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b85eea2ad540445dbd95a655135dc043
I20250110 02:51:10.042414 1178 ts_tablet_manager.cc:1939] T b85eea2ad540445dbd95a655135dc043 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.043084 918 raft_consensus.cc:2463] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:10.043581 994 leader_election.cc:304] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:10.043897 1258 raft_consensus.cc:2798] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:10.044071 1258 raft_consensus.cc:695] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:10.044447 1258 consensus_queue.cc:237] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.045612 1212 raft_consensus.cc:491] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.045795 1212 raft_consensus.cc:513] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.046720 1212 leader_election.cc:290] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.047530 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c845c27a86d4e4f93c932313ebf648b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:10.048326 709 leader_election.cc:343] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.048851 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c845c27a86d4e4f93c932313ebf648b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:10.049119 1059 raft_consensus.cc:2463] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:10.049590 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 7422cfed02fc4b50ab867fdf34769b58
I20250110 02:51:10.050153 521 catalog_manager.cc:6596] ReplaceTablet: tablet 7422cfed02fc4b50ab867fdf34769b58 deleted and replaced by tablet 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:10.051743 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 with delete_type TABLET_DATA_DELETED (replaced by tablet 344184e1c98d422095cb254f5b2bc792) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.052801 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 with delete_type TABLET_DATA_DELETED (replaced by tablet 344184e1c98d422095cb254f5b2bc792) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.052866 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7422cfed02fc4b50ab867fdf34769b58 with delete_type TABLET_DATA_DELETED (replaced by tablet 344184e1c98d422095cb254f5b2bc792) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.053313 711 leader_election.cc:304] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:10.053810 1212 raft_consensus.cc:2798] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:10.053946 1212 raft_consensus.cc:491] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:10.054042 1212 raft_consensus.cc:3054] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.060281 1212 raft_consensus.cc:513] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.061343 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c845c27a86d4e4f93c932313ebf648b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:10.061534 1059 raft_consensus.cc:3054] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.063755 776 raft_consensus.cc:2463] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:10.064234 1059 raft_consensus.cc:2463] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:10.064683 711 leader_election.cc:304] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:10.064999 1187 raft_consensus.cc:2798] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:10.065686 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d9338439b43445e9b19ed1f53c53d149"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.066442 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9338439b43445e9b19ed1f53c53d149 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.071440 1212 leader_election.cc:290] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.073101 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c845c27a86d4e4f93c932313ebf648b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:10.073817 709 leader_election.cc:343] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.076944 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b85eea2ad540445dbd95a655135dc043 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
I20250110 02:51:10.077970 1188 tablet_bootstrap.cc:492] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.078182 1188 ts_tablet_manager.cc:1397] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.084s user 0.020s sys 0.012s
W20250110 02:51:10.078663 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1aafff2e0de548129d99e9f07dc3ab11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aafff2e0de548129d99e9f07dc3ab11 already in progress: creating tablet
W20250110 02:51:10.079452 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 02d3f60c819f4824bc99d755e7442976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d3f60c819f4824bc99d755e7442976 already in progress: creating tablet
I20250110 02:51:10.079452 1188 raft_consensus.cc:357] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.079274 1279 leader_election.cc:290] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.079677 1188 raft_consensus.cc:383] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.079778 1188 raft_consensus.cc:738] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.080054 1188 consensus_queue.cc:260] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
W20250110 02:51:10.080420 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 886ab6d12cf545b9b700959e18396c50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886ab6d12cf545b9b700959e18396c50 already in progress: creating tablet
I20250110 02:51:10.080631 1187 raft_consensus.cc:695] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:10.081022 1187 consensus_queue.cc:237] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.081329 1188 ts_tablet_manager.cc:1428] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:10.082556 1188 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.083356 1310 raft_consensus.cc:491] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:10.083559 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 10ab917bde534e3fbdf5dbae4af9304e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10ab917bde534e3fbdf5dbae4af9304e already in progress: creating tablet
I20250110 02:51:10.083503 1310 raft_consensus.cc:513] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.084618 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80900124f93d43a0abf93175b16225a7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:10.084955 1059 raft_consensus.cc:2463] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:10.085453 711 leader_election.cc:304] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:10.085819 1212 raft_consensus.cc:2798] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:10.085958 1212 raft_consensus.cc:491] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:10.086110 1212 raft_consensus.cc:3054] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.087074 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_bfd353801c504f1aab972805a7301c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.087702 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfd353801c504f1aab972805a7301c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.088681 1212 raft_consensus.cc:513] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.088922 1188 tablet_bootstrap.cc:654] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.089731 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80900124f93d43a0abf93175b16225a7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:10.089939 1059 raft_consensus.cc:3054] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.092219 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7ba31aa0d79e4bea8e9407de8ddf6d14" } new_table_name: "default.Table_459a9c14d71c40e5a90a716b251f8be4" modify_external_catalogs: true
I20250110 02:51:10.092599 1059 raft_consensus.cc:2463] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:10.093034 711 leader_election.cc:304] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:10.093477 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8c550622172349e2afb560f02401b8fa" } new_table_name: "default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5" modify_external_catalogs: true
I20250110 02:51:10.094560 1178 tablet_replica.cc:331] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.098526 1178 raft_consensus.cc:2238] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.099474 1212 leader_election.cc:290] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.100191 1212 raft_consensus.cc:2798] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:10.100687 1212 raft_consensus.cc:695] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:10.100891 1178 raft_consensus.cc:2267] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.101253 1212 consensus_queue.cc:237] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.106390 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 9bce8f5888bb404696e46487176b5fe8
I20250110 02:51:10.106657 1310 leader_election.cc:290] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.107024 519 catalog_manager.cc:6596] ReplaceTablet: tablet 9bce8f5888bb404696e46487176b5fe8 deleted and replaced by tablet 3934e6611d4e4b94ba5185f33c3d05f4
I20250110 02:51:10.107492 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7a06870649f8422cb01b67374a80956c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.108235 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a06870649f8422cb01b67374a80956c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.111624 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bce8f5888bb404696e46487176b5fe8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3934e6611d4e4b94ba5185f33c3d05f4) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.111621 1178 ts_tablet_manager.cc:1905] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.112247 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bce8f5888bb404696e46487176b5fe8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3934e6611d4e4b94ba5185f33c3d05f4) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.113657 1314 raft_consensus.cc:491] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.114452 1314 raft_consensus.cc:513] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.117683 1314 leader_election.cc:290] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:10.118434 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:10.119099 711 leader_election.cc:343] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.120862 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6365c4bf56b44696a42c21a809c8210c" } modify_external_catalogs: true
I20250110 02:51:10.121088 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6365c4bf56b44696a42c21a809c8210c" } modify_external_catalogs: true
I20250110 02:51:10.130792 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f86542fc90134a57b994da74f3e5b461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.138710 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f86542fc90134a57b994da74f3e5b461 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:10.145182 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d98b789eaabe4191af40c9f713acb030 consensus state has no leader
I20250110 02:51:10.147495 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f0abf378d8b94be9b42028b3ef26bbe5" } modify_external_catalogs: true
I20250110 02:51:10.147687 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f0abf378d8b94be9b42028b3ef26bbe5" } modify_external_catalogs: true
I20250110 02:51:10.154742 1179 tablet_replica.cc:331] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:10.155138 1179 raft_consensus.cc:2238] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.155403 1179 raft_consensus.cc:2267] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.156206 1179 ts_tablet_manager.cc:1905] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:10.164309 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3934e6611d4e4b94ba5185f33c3d05f4 has no consensus state
I20250110 02:51:10.164649 523 master_service.cc:946] ReplaceTablet: received request to replace tablet 10504e77a5c64ebf85be937d64ca9c80 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.165566 1180 ts_tablet_manager.cc:1918] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.168316 1180 log.cc:1198] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1c46be1162124b50ae43d8ea57cb5a0b
I20250110 02:51:10.173324 1307 tablet.cc:1721] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:10.175386 1308 tablet.cc:1721] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:10.177446 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bce8f5888bb404696e46487176b5fe8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3934e6611d4e4b94ba5185f33c3d05f4) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:10.179430 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 80075.3us
I20250110 02:51:10.180197 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80900124f93d43a0abf93175b16225a7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:10.180387 917 raft_consensus.cc:3054] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.180954 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80900124f93d43a0abf93175b16225a7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.183017 917 raft_consensus.cc:2463] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:10.183429 1304 mvcc.cc:204] Tried to move back new op lower bound from 7112611716893949952 to 7112611714836803584. Current Snapshot: MvccSnapshot[applied={T|T < 7112611716893949952}]
I20250110 02:51:10.185318 1180 ts_tablet_manager.cc:1939] T 1c46be1162124b50ae43d8ea57cb5a0b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.188318 1180 tablet_replica.cc:331] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.188668 1180 raft_consensus.cc:2238] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:10.189851 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.190568 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1c46be1162124b50ae43d8ea57cb5a0b (table default.Table_f97a80832b8348cc9f5839ebe6887044 [id=1a1e7395b41a4c9cae24c80f746dff63]) successfully deleted
I20250110 02:51:10.202173 1180 pending_rounds.cc:62] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:10.202353 1180 pending_rounds.cc:69] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611715732451328 op_type: NO_OP noop_request { }
I20250110 02:51:10.203276 1180 raft_consensus.cc:2883] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:10.203408 1180 raft_consensus.cc:2267] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:10.204330 709 leader_election.cc:343] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.204496 709 leader_election.cc:304] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:10.204952 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b691c5c8267948a08b5a76dc5675aa52" } new_table_name: "default.Table_7cebc36aba3d43e0877447c797c945f7" modify_external_catalogs: true
I20250110 02:51:10.205286 1315 raft_consensus.cc:2743] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.207377 1180 ts_tablet_manager.cc:1905] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.223385 1184 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:10.223635 1184 ts_tablet_manager.cc:1397] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.240s user 0.008s sys 0.064s
I20250110 02:51:10.224654 1184 raft_consensus.cc:357] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.224949 1184 raft_consensus.cc:383] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.225096 1184 raft_consensus.cc:738] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.225410 1184 consensus_queue.cc:260] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.227295 1184 ts_tablet_manager.cc:1428] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:10.227685 1184 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:10.228320 1181 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.228487 1181 ts_tablet_manager.cc:1397] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.247s user 0.015s sys 0.049s
I20250110 02:51:10.229213 1181 raft_consensus.cc:357] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.229454 1181 raft_consensus.cc:383] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.229588 1181 raft_consensus.cc:738] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.229907 1181 consensus_queue.cc:260] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.230556 1181 ts_tablet_manager.cc:1428] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:10.231729 1181 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.233386 1181 tablet_bootstrap.cc:654] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.233824 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d24028774b004dbfabe940b74a12bd16" } modify_external_catalogs: true
I20250110 02:51:10.234045 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d24028774b004dbfabe940b74a12bd16" } modify_external_catalogs: true
I20250110 02:51:10.237648 1320 tablet.cc:1716] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Skipping requested alter to schema version 2, tablet already version 2
I20250110 02:51:10.239936 1319 tablet.cc:1721] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:10.242069 1184 tablet_bootstrap.cc:654] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.234498 1180 ts_tablet_manager.cc:1918] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.244349 1180 log.cc:1198] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/286aee6074e14c27b17bc79a78906620
I20250110 02:51:10.244894 1180 ts_tablet_manager.cc:1939] T 286aee6074e14c27b17bc79a78906620 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.245064 1178 ts_tablet_manager.cc:1918] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.245901 1180 tablet_replica.cc:331] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.245859 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 286aee6074e14c27b17bc79a78906620 (table default.Table_ced8b781ad0f4d948f737825145e4dbf [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
I20250110 02:51:10.246174 1180 raft_consensus.cc:2238] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.246433 1180 raft_consensus.cc:2267] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.250859 1308 tablet.cc:1716] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Skipping requested alter to schema version 2, tablet already version 2
I20250110 02:51:10.253625 1180 ts_tablet_manager.cc:1905] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.257220 1179 ts_tablet_manager.cc:1918] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.258481 524 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 10504e77a5c64ebf85be937d64ca9c80 (table default.Table_1a58fb7be3a34571bedcfbbff5db8ea7 [id=dd4008ea14ad40c68b3e71fac9e2be88]). Expected version 2 got 0
I20250110 02:51:10.258773 524 catalog_manager.cc:5526] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:10.259171 524 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]). Expected version 1 got 0
I20250110 02:51:10.259407 524 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]). Expected version 1 got 0
W20250110 02:51:10.260185 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe86be26d1b544c08b231f20085d3159 consensus state has no leader
I20250110 02:51:10.260424 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c9d791b1c504171ba66dec55c8fb0e6 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.260735 520 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 825b4d370dab4b908ddde0f1dddf466f
I20250110 02:51:10.263058 751 tablet_service.cc:1467] Processing CreateTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb (DEFAULT_TABLE table=default.Table_4813a1a74b1a455584f21646ec5a97d6 [id=d7bbf5b39b224708974b5f6da6429fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.263382 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 (DEFAULT_TABLE table=default.Table_bcc7554985664d30b2fca81e32af5394 [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.263458 1181 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.267218 756 tablet_service.cc:1467] Processing CreateTablet for tablet 423c820b81b845719d59a376c18e175a (DEFAULT_TABLE table=default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.267643 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c77ec8c7724d5fb408a8bfa53811eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.267750 1181 ts_tablet_manager.cc:1397] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20250110 02:51:10.267609 1179 log.cc:1198] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7422cfed02fc4b50ab867fdf34769b58
I20250110 02:51:10.267939 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b74812cb81c44fcb8786cc5dae328e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.268343 1179 ts_tablet_manager.cc:1939] T 7422cfed02fc4b50ab867fdf34769b58 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:10.268388 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423c820b81b845719d59a376c18e175a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.268399 752 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc0532c94eb401e9d299e79556b4cf1 (DEFAULT_TABLE table=default.Table_78653a7a5f0743f18b98b3e703c05a6c [id=8210fc764873462ca2238d58c2b61bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.268843 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc0532c94eb401e9d299e79556b4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.269366 1179 tablet_replica.cc:331] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:10.269567 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb (DEFAULT_TABLE table=default.Table_4813a1a74b1a455584f21646ec5a97d6 [id=d7bbf5b39b224708974b5f6da6429fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.269904 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c77ec8c7724d5fb408a8bfa53811eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.270260 1179 raft_consensus.cc:2238] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.270534 1179 raft_consensus.cc:2267] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.271070 1179 ts_tablet_manager.cc:1905] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.272490 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7422cfed02fc4b50ab867fdf34769b58 (table default.Table_b586408fcd3741a3a2b249e88f54eacf [id=141731490a99451e9f722fcc3d5857bd]) successfully deleted
I20250110 02:51:10.273497 1184 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:10.273850 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 423c820b81b845719d59a376c18e175a (DEFAULT_TABLE table=default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.274547 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423c820b81b845719d59a376c18e175a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.274704 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c01af512f460daa98b19444f72172 (DEFAULT_TABLE table=default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.276934 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 (DEFAULT_TABLE table=default.Table_bcc7554985664d30b2fca81e32af5394 [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.277241 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2c01af512f460daa98b19444f72172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.276947 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc0532c94eb401e9d299e79556b4cf1 (DEFAULT_TABLE table=default.Table_78653a7a5f0743f18b98b3e703c05a6c [id=8210fc764873462ca2238d58c2b61bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.281184 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 80900124f93d43a0abf93175b16225a7 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.281484 1039 tablet_service.cc:1467] Processing CreateTablet for tablet d5f56778a0244378ad8499a4ceef3510 (DEFAULT_TABLE table=default.Table_a9587fbd6e924c49b1a9a1189bff5d63 [id=4e31223c509f46eba782e7c34f293c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.281626 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3edf95be0d264f409133f83c3e948d58" } modify_external_catalogs: true
I20250110 02:51:10.281781 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 00618f611d4c4a84a988125c5d175efc (DEFAULT_TABLE table=default.Table_2500bc74e2d44275bab206e0ba400d75 [id=fda1f290a7fb4b46a24125ae643d52b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.281823 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3edf95be0d264f409133f83c3e948d58" } modify_external_catalogs: true
I20250110 02:51:10.282516 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc0532c94eb401e9d299e79556b4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.284698 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c2412420dc54258a6921ead194c1587 consensus state has no leader
I20250110 02:51:10.285410 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1c2495b680684279ae633a18f91a091e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.285866 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2495b680684279ae633a18f91a091e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.286514 1036 tablet_service.cc:1467] Processing CreateTablet for tablet fe86be26d1b544c08b231f20085d3159 (DEFAULT_TABLE table=default.Table_b439c40f07fd4b48b1735826b7eedbf0 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.286118 1181 raft_consensus.cc:357] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.288857 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_59d402904b724493ba7a5f2a0710b283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.289083 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c683bc3feaff44b1830cccfe839f677e
W20250110 02:51:10.289305 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59d402904b724493ba7a5f2a0710b283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.289753 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.290172 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.290591 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00618f611d4c4a84a988125c5d175efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.292107 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f56778a0244378ad8499a4ceef3510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.292196 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7a06870649f8422cb01b67374a80956c" } new_table_name: "default.Table_163c6a20e5ad4f8398dc3a947c92bf9f" modify_external_catalogs: true
I20250110 02:51:10.282166 1035 tablet_service.cc:1467] Processing CreateTablet for tablet c65704c9d486444ebe8e4ffd4580f395 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.292484 1038 tablet_service.cc:1467] Processing CreateTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (DEFAULT_TABLE table=default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.292560 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe86be26d1b544c08b231f20085d3159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.292701 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ad9f920cde954f31acebc6d0b6965e58 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.293115 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet c683bc3feaff44b1830cccfe839f677e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.293388 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:10.293958 521 master_service.cc:946] ReplaceTablet: received request to replace tablet aa40d20288644b2a8d36ede967d16e79 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.296847 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 80900124f93d43a0abf93175b16225a7
I20250110 02:51:10.297297 517 catalog_manager.cc:6596] ReplaceTablet: tablet 80900124f93d43a0abf93175b16225a7 deleted and replaced by tablet 019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:10.298662 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:10.299340 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0d23d32f6ce84a0594f62a4ca18dcc62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.299750 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.302244 1181 raft_consensus.cc:383] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.302783 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ce1eaf1a53cd4620a73549f2734f168b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.303246 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b74812cb81c44fcb8786cc5dae328e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.303267 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce1eaf1a53cd4620a73549f2734f168b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.305176 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.304510 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 514704cd935b44e8b760221224332509 (DEFAULT_TABLE table=default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.306123 1181 raft_consensus.cc:738] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.306757 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 (DEFAULT_TABLE table=default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.307221 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9dbb61a51f14a3f96cc3c77981e8ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.307366 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d58e92b9789e49f6bbfbefe3f8f3a0ef
I20250110 02:51:10.308126 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_17a3ad3c46d2416db2b8bd0f66e39703"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.308341 1181 consensus_queue.cc:260] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:10.308506 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17a3ad3c46d2416db2b8bd0f66e39703 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.310570 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.311486 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.312474 1184 ts_tablet_manager.cc:1397] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.009s
I20250110 02:51:10.313364 1184 raft_consensus.cc:357] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.313469 754 tablet_service.cc:1467] Processing CreateTablet for tablet d5f56778a0244378ad8499a4ceef3510 (DEFAULT_TABLE table=default.Table_a9587fbd6e924c49b1a9a1189bff5d63 [id=4e31223c509f46eba782e7c34f293c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.313606 1184 raft_consensus.cc:383] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.313699 1184 raft_consensus.cc:738] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.313838 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f56778a0244378ad8499a4ceef3510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.314034 1184 consensus_queue.cc:260] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.314754 1184 ts_tablet_manager.cc:1428] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:10.314997 1184 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:10.315623 1181 ts_tablet_manager.cc:1428] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20250110 02:51:10.316236 750 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c01af512f460daa98b19444f72172 (DEFAULT_TABLE table=default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.317085 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 80900124f93d43a0abf93175b16225a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 019c0d60da5548bb86c93ccd89bf94ff) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.317595 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c845c27a86d4e4f93c932313ebf648b from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.317648 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514704cd935b44e8b760221224332509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.318764 1178 log.cc:1198] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/24efc81a5ac743809665b17f941c0652
I20250110 02:51:10.319152 1178 ts_tablet_manager.cc:1939] T 24efc81a5ac743809665b17f941c0652 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.319289 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.319624 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.320195 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 10504e77a5c64ebf85be937d64ca9c80
I20250110 02:51:10.320281 1178 tablet_replica.cc:331] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.320459 1178 raft_consensus.cc:2238] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:10.320639 523 catalog_manager.cc:6596] ReplaceTablet: tablet 10504e77a5c64ebf85be937d64ca9c80 deleted and replaced by tablet c4ddce8ad8234c6cb20e6015e489a340
I20250110 02:51:10.320822 1178 pending_rounds.cc:62] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Trying to abort 1 pending ops.
I20250110 02:51:10.320977 1178 pending_rounds.cc:69] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611716901367808 op_type: NO_OP noop_request { }
I20250110 02:51:10.321197 1178 raft_consensus.cc:2883] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:10.321367 1178 raft_consensus.cc:2267] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.262504 1188 tablet_bootstrap.cc:492] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.321929 517 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]). Expected version 1 got 0
I20250110 02:51:10.275281 895 tablet_service.cc:1467] Processing CreateTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb (DEFAULT_TABLE table=default.Table_4813a1a74b1a455584f21646ec5a97d6 [id=d7bbf5b39b224708974b5f6da6429fe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.322139 1188 ts_tablet_manager.cc:1397] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.240s user 0.047s sys 0.099s
I20250110 02:51:10.275568 894 tablet_service.cc:1467] Processing CreateTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 (DEFAULT_TABLE table=default.Table_bcc7554985664d30b2fca81e32af5394 [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.275756 897 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2c01af512f460daa98b19444f72172 (DEFAULT_TABLE table=default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.322335 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c77ec8c7724d5fb408a8bfa53811eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.322602 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2c01af512f460daa98b19444f72172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.324625 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 24efc81a5ac743809665b17f941c0652 (table default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]) successfully deleted
W20250110 02:51:10.324955 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4c9d791b1c504171ba66dec55c8fb0e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9d791b1c504171ba66dec55c8fb0e6 already in progress: creating tablet
I20250110 02:51:10.325134 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 with delete_type TABLET_DATA_DELETED (replaced by tablet c4ddce8ad8234c6cb20e6015e489a340) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.276074 890 tablet_service.cc:1467] Processing CreateTablet for tablet 00618f611d4c4a84a988125c5d175efc (DEFAULT_TABLE table=default.Table_2500bc74e2d44275bab206e0ba400d75 [id=fda1f290a7fb4b46a24125ae643d52b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.276118 896 tablet_service.cc:1467] Processing CreateTablet for tablet d5f56778a0244378ad8499a4ceef3510 (DEFAULT_TABLE table=default.Table_a9587fbd6e924c49b1a9a1189bff5d63 [id=4e31223c509f46eba782e7c34f293c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.276083 891 tablet_service.cc:1467] Processing CreateTablet for tablet 423c820b81b845719d59a376c18e175a (DEFAULT_TABLE table=default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.276326 898 tablet_service.cc:1467] Processing CreateTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (DEFAULT_TABLE table=default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.326701 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00618f611d4c4a84a988125c5d175efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.326897 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f56778a0244378ad8499a4ceef3510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.326910 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a38779a8c72a4768a0c2adba3eb2f017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.327196 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9dbb61a51f14a3f96cc3c77981e8ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.276764 892 tablet_service.cc:1467] Processing CreateTablet for tablet fe86be26d1b544c08b231f20085d3159 (DEFAULT_TABLE table=default.Table_b439c40f07fd4b48b1735826b7eedbf0 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:10.327433 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a38779a8c72a4768a0c2adba3eb2f017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.327849 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe86be26d1b544c08b231f20085d3159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.278869 889 tablet_service.cc:1467] Processing CreateTablet for tablet c65704c9d486444ebe8e4ffd4580f395 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.279459 888 tablet_service.cc:1467] Processing CreateTablet for tablet 514704cd935b44e8b760221224332509 (DEFAULT_TABLE table=default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.330703 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65704c9d486444ebe8e4ffd4580f395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.330994 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514704cd935b44e8b760221224332509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.280169 887 tablet_service.cc:1467] Processing CreateTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 (DEFAULT_TABLE table=default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.331204 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b74812cb81c44fcb8786cc5dae328e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.280520 886 tablet_service.cc:1467] Processing CreateTablet for tablet ad9f920cde954f31acebc6d0b6965e58 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.331569 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70af6ec40c7a4442b91680d8125c2d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.332636 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65704c9d486444ebe8e4ffd4580f395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.333356 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9f920cde954f31acebc6d0b6965e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.333386 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 286aee6074e14c27b17bc79a78906620 with delete_type TABLET_DATA_DELETED (replaced by tablet fe86be26d1b544c08b231f20085d3159) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.332733 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.333333 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 with delete_type TABLET_DATA_DELETED (replaced by tablet c4ddce8ad8234c6cb20e6015e489a340) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.334012 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70af6ec40c7a4442b91680d8125c2d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.334467 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423c820b81b845719d59a376c18e175a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.334810 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfc062b5fcb42c3bce6dfaed54b3869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.284763 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.295501 884 tablet_service.cc:1514] Processing DeleteTablet for tablet c683bc3feaff44b1830cccfe839f677e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.295729 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.299520 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 80900124f93d43a0abf93175b16225a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 019c0d60da5548bb86c93ccd89bf94ff) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.301498 881 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.305701 1324 tablet.cc:1716] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Skipping requested alter to schema version 2, tablet already version 2
I20250110 02:51:10.275607 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc0532c94eb401e9d299e79556b4cf1 (DEFAULT_TABLE table=default.Table_78653a7a5f0743f18b98b3e703c05a6c [id=8210fc764873462ca2238d58c2b61bfb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:10.338542 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48cb20d47ace4fd49f9dc4e526cab702 has no consensus state
I20250110 02:51:10.338937 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet aa40d20288644b2a8d36ede967d16e79
I20250110 02:51:10.339391 521 catalog_manager.cc:6596] ReplaceTablet: tablet aa40d20288644b2a8d36ede967d16e79 deleted and replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:10.341279 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2c01af512f460daa98b19444f72172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.342403 749 tablet_service.cc:1467] Processing CreateTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (DEFAULT_TABLE table=default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.343973 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_215c0fc0ef884696ad73db0829ca7e67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.344456 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_215c0fc0ef884696ad73db0829ca7e67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.342301 755 tablet_service.cc:1467] Processing CreateTablet for tablet 00618f611d4c4a84a988125c5d175efc (DEFAULT_TABLE table=default.Table_2500bc74e2d44275bab206e0ba400d75 [id=fda1f290a7fb4b46a24125ae643d52b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.346211 1181 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.347075 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_335d6489a4484f1fb0de9408d54cfab9" } new_table_name: "default.Table_e55e81b07bfe4a668255066cb6447735" modify_external_catalogs: true
I20250110 02:51:10.347376 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9dbb61a51f14a3f96cc3c77981e8ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.348647 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 4c845c27a86d4e4f93c932313ebf648b
I20250110 02:51:10.349112 524 catalog_manager.cc:6596] ReplaceTablet: tablet 4c845c27a86d4e4f93c932313ebf648b deleted and replaced by tablet 7b82be36e1754d62894c00ed29db227f
I20250110 02:51:10.350486 753 tablet_service.cc:1467] Processing CreateTablet for tablet fe86be26d1b544c08b231f20085d3159 (DEFAULT_TABLE table=default.Table_b439c40f07fd4b48b1735826b7eedbf0 [id=621d3211bd1649a0aab67ae4c000f40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.355051 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c845c27a86d4e4f93c932313ebf648b with delete_type TABLET_DATA_DELETED (replaced by tablet 7b82be36e1754d62894c00ed29db227f) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.355474 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.356592 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc0532c94eb401e9d299e79556b4cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.358333 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00618f611d4c4a84a988125c5d175efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.359658 1315 raft_consensus.cc:491] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.362111 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe86be26d1b544c08b231f20085d3159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.352770 884 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.362509 1178 ts_tablet_manager.cc:1905] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.363772 1315 raft_consensus.cc:513] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.365370 746 tablet_service.cc:1467] Processing CreateTablet for tablet 514704cd935b44e8b760221224332509 (DEFAULT_TABLE table=default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.367909 744 tablet_service.cc:1467] Processing CreateTablet for tablet ad9f920cde954f31acebc6d0b6965e58 (DEFAULT_TABLE table=default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.369009 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfc062b5fcb42c3bce6dfaed54b3869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.369468 517 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a] alter complete (version 2)
I20250110 02:51:10.375226 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514704cd935b44e8b760221224332509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.366423 747 tablet_service.cc:1467] Processing CreateTablet for tablet c65704c9d486444ebe8e4ffd4580f395 (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.377040 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4813a1a74b1a455584f21646ec5a97d6" } new_table_name: "default.Table_e676f1148b5f4988a2be33981eec57c3" modify_external_catalogs: true
I20250110 02:51:10.384313 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:10.386829 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 has no consensus state
I20250110 02:51:10.387017 1188 raft_consensus.cc:357] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.387225 1188 raft_consensus.cc:383] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.388018 1188 raft_consensus.cc:738] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.388294 1188 consensus_queue.cc:260] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.389658 1188 ts_tablet_manager.cc:1428] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20250110 02:51:10.389972 1188 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.390479 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c845c27a86d4e4f93c932313ebf648b with delete_type TABLET_DATA_DELETED (replaced by tablet 7b82be36e1754d62894c00ed29db227f) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:10.391141 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64c77ec8c7724d5fb408a8bfa53811eb consensus state has no leader
I20250110 02:51:10.393738 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9f920cde954f31acebc6d0b6965e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.396071 1179 ts_tablet_manager.cc:1918] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.396207 1179 log.cc:1198] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/9bce8f5888bb404696e46487176b5fe8
I20250110 02:51:10.399367 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_87db4b399a114ebd9397992a26db0aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.399844 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87db4b399a114ebd9397992a26db0aba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.401129 1184 tablet_bootstrap.cc:654] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.402588 745 tablet_service.cc:1467] Processing CreateTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 (DEFAULT_TABLE table=default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.409677 742 tablet_service.cc:1514] Processing DeleteTablet for tablet c683bc3feaff44b1830cccfe839f677e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.411556 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 286aee6074e14c27b17bc79a78906620 with delete_type TABLET_DATA_DELETED (replaced by tablet fe86be26d1b544c08b231f20085d3159) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.412377 1181 tablet_bootstrap.cc:654] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.413803 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.413856 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d5f9bda55942ff856252bbb50b3c5b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:10.414644 510 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 66f6f4f221e6445db0bb569f372cfaa0 (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 66f6f4f221e6445db0bb569f372cfaa0 consensus state has no leader
I20250110 02:51:10.416189 1315 leader_election.cc:290] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.417594 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9f920cde954f31acebc6d0b6965e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.417865 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70af6ec40c7a4442b91680d8125c2d48. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.418718 709 leader_election.cc:343] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.419427 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 89984fa8f39641a39969f4e3392ae8b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.424760 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b851ddf27fe44d19b9ca39d82e6352cd with delete_type TABLET_DATA_DELETED (replaced by tablet 66f6f4f221e6445db0bb569f372cfaa0) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.427012 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfc062b5fcb42c3bce6dfaed54b3869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.428076 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_eb30b4d9c27e4b5788054f7205e5610a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.429003 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb30b4d9c27e4b5788054f7205e5610a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.431083 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 80900124f93d43a0abf93175b16225a7 with delete_type TABLET_DATA_DELETED (replaced by tablet 019c0d60da5548bb86c93ccd89bf94ff) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.434401 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.441735 1188 tablet_bootstrap.cc:654] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.442973 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65704c9d486444ebe8e4ffd4580f395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.448978 1188 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.449159 1188 ts_tablet_manager.cc:1397] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.003s
I20250110 02:51:10.449831 1188 raft_consensus.cc:357] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.450135 1188 raft_consensus.cc:383] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.450232 1188 raft_consensus.cc:738] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.450527 1188 consensus_queue.cc:260] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.453812 1188 ts_tablet_manager.cc:1428] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:10.455351 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 886ab6d12cf545b9b700959e18396c50 with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d4c4631994a5792200ba4d4c906f0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.456053 1188 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.457441 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c845c27a86d4e4f93c932313ebf648b with delete_type TABLET_DATA_DELETED (replaced by tablet 7b82be36e1754d62894c00ed29db227f) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.458429 1188 tablet_bootstrap.cc:654] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.456267 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.463373 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_88a8376ca7864049a601575865aa9b88" } new_table_name: "default.Table_15356bd1a2bd4846be42885b635b99ad" modify_external_catalogs: true
I20250110 02:51:10.463519 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1c2495b680684279ae633a18f91a091e" } modify_external_catalogs: true
I20250110 02:51:10.463709 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1c2495b680684279ae633a18f91a091e" } modify_external_catalogs: true
I20250110 02:51:10.461975 742 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.470300 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d5f9bda55942ff856252bbb50b3c5b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:10.470535 1060 raft_consensus.cc:2463] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:10.472999 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 10504e77a5c64ebf85be937d64ca9c80 with delete_type TABLET_DATA_DELETED (replaced by tablet c4ddce8ad8234c6cb20e6015e489a340) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.475836 1180 ts_tablet_manager.cc:1918] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.475981 1180 log.cc:1198] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:10.476392 1180 ts_tablet_manager.cc:1939] T 2a27f0327d1844b8a561af8b0bf4385b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.477532 1180 tablet_replica.cc:331] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.477761 1180 raft_consensus.cc:2238] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.477958 1180 raft_consensus.cc:2267] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.482354 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2a27f0327d1844b8a561af8b0bf4385b (table default.Table_cbaca8b1d598406db88d9fcef46dd17b [id=b74f16dd5b58413c8dff82451dd83cd6]) successfully deleted
W20250110 02:51:10.482434 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67adef4c3f4041e1ab480100ca762f62 consensus state has no leader
I20250110 02:51:10.483307 1180 ts_tablet_manager.cc:1905] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.483616 1188 tablet_bootstrap.cc:492] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.483775 1188 ts_tablet_manager.cc:1397] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20250110 02:51:10.484447 1188 raft_consensus.cc:357] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.484645 1188 raft_consensus.cc:383] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.484738 1188 raft_consensus.cc:738] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.484928 1188 consensus_queue.cc:260] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.487679 1188 ts_tablet_manager.cc:1428] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:10.487960 1188 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.488444 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b586408fcd3741a3a2b249e88f54eacf" } new_table_name: "default.Table_196d49d7fc844b019f3a7356659f5b72" modify_external_catalogs: true
I20250110 02:51:10.489655 1188 tablet_bootstrap.cc:654] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.492897 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 1dfc062b5fcb42c3bce6dfaed54b3869 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.493891 1188 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.495076 1188 ts_tablet_manager.cc:1397] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
W20250110 02:51:10.496075 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 344184e1c98d422095cb254f5b2bc792 has no consensus state
I20250110 02:51:10.496003 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_33f285fec4b94dc38471bf0dbf5fb92b" } modify_external_catalogs: true
I20250110 02:51:10.496187 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_33f285fec4b94dc38471bf0dbf5fb92b" } modify_external_catalogs: true
I20250110 02:51:10.496448 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a38779a8c72a4768a0c2adba3eb2f017" } modify_external_catalogs: true
I20250110 02:51:10.495741 1188 raft_consensus.cc:357] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.496953 1188 raft_consensus.cc:383] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.497133 1188 raft_consensus.cc:738] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.497617 1188 consensus_queue.cc:260] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.498888 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a38779a8c72a4768a0c2adba3eb2f017" } modify_external_catalogs: true
I20250110 02:51:10.499984 1188 ts_tablet_manager.cc:1428] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250110 02:51:10.500316 1188 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.503538 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7c26cb6440db4eb8a73ac05566a26a75" } modify_external_catalogs: true
I20250110 02:51:10.503733 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7c26cb6440db4eb8a73ac05566a26a75" } modify_external_catalogs: true
I20250110 02:51:10.504855 1198 raft_consensus.cc:491] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.504981 1198 raft_consensus.cc:513] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.505474 1188 tablet_bootstrap.cc:654] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.505635 1198 leader_election.cc:290] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.506600 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "886ab6d12cf545b9b700959e18396c50" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:10.507141 853 leader_election.cc:343] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.509490 525 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]). Expected version 1 got 0
I20250110 02:51:10.510205 886 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 (DEFAULT_TABLE table=default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.510610 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb20d47ace4fd49f9dc4e526cab702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.513707 1188 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.513952 1188 ts_tablet_manager.cc:1397] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20250110 02:51:10.514307 521 master_service.cc:946] ReplaceTablet: received request to replace tablet c65704c9d486444ebe8e4ffd4580f395 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.514463 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9c251b39e0f04bf7916d2ce12652063c" } modify_external_catalogs: true
I20250110 02:51:10.514623 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9c251b39e0f04bf7916d2ce12652063c" } modify_external_catalogs: true
I20250110 02:51:10.515411 1180 ts_tablet_manager.cc:1918] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.515517 1180 log.cc:1198] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/89984fa8f39641a39969f4e3392ae8b7
I20250110 02:51:10.515882 1180 ts_tablet_manager.cc:1939] T 89984fa8f39641a39969f4e3392ae8b7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.516462 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_18721d8cd9034ade8df5d1035133a6a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.516660 1188 raft_consensus.cc:357] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.516822 1188 raft_consensus.cc:383] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.516912 1188 raft_consensus.cc:738] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:10.516988 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18721d8cd9034ade8df5d1035133a6a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.517169 1188 consensus_queue.cc:260] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.517673 1188 ts_tablet_manager.cc:1428] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:10.517742 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 89984fa8f39641a39969f4e3392ae8b7 (table default.Table_46bcf45f1fce419c882b8c239e680ee9 [id=1da07435b54849318beda7e8518f0bf6]) successfully deleted
I20250110 02:51:10.519030 1188 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.518976 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 (DEFAULT_TABLE table=default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.520682 1188 tablet_bootstrap.cc:654] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.521936 519 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7b82be36e1754d62894c00ed29db227f
I20250110 02:51:10.521973 524 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet fa543ad7fd14401e8d8d1da14e7abd86
W20250110 02:51:10.522228 853 leader_election.cc:343] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 886ab6d12cf545b9b700959e18396c50
I20250110 02:51:10.522348 853 leader_election.cc:304] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.523459 1198 raft_consensus.cc:2743] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.523913 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb20d47ace4fd49f9dc4e526cab702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.533015 519 master_service.cc:946] ReplaceTablet: received request to replace tablet a9f404a2e7494645af449aea3d3d3bf0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.533774 1315 raft_consensus.cc:491] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.533928 1315 raft_consensus.cc:513] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.535143 1178 ts_tablet_manager.cc:1918] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.535254 1178 log.cc:1198] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7422cfed02fc4b50ab867fdf34769b58
I20250110 02:51:10.535696 1178 ts_tablet_manager.cc:1939] T 7422cfed02fc4b50ab867fdf34769b58 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.536406 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.536576 1178 tablet_replica.cc:331] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.536785 1178 raft_consensus.cc:2238] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.536816 1315 leader_election.cc:290] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.536945 1178 raft_consensus.cc:2267] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:10.538082 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a9f404a2e7494645af449aea3d3d3bf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f404a2e7494645af449aea3d3d3bf0 already in progress: creating tablet
I20250110 02:51:10.538146 1180 tablet_replica.cc:331] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.538311 1180 raft_consensus.cc:2238] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.538501 1180 raft_consensus.cc:2267] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:10.538491 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1aafff2e0de548129d99e9f07dc3ab11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aafff2e0de548129d99e9f07dc3ab11 already in progress: creating tablet
W20250110 02:51:10.538725 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 02d3f60c819f4824bc99d755e7442976 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d3f60c819f4824bc99d755e7442976 already in progress: creating tablet
I20250110 02:51:10.539024 1180 ts_tablet_manager.cc:1905] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.539139 1178 ts_tablet_manager.cc:1905] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:10.539433 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 886ab6d12cf545b9b700959e18396c50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886ab6d12cf545b9b700959e18396c50 already in progress: creating tablet
W20250110 02:51:10.539649 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 10ab917bde534e3fbdf5dbae4af9304e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10ab917bde534e3fbdf5dbae4af9304e already in progress: creating tablet
I20250110 02:51:10.534642 711 leader_election.cc:304] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:10.541605 754 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 (DEFAULT_TABLE table=default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.542150 1184 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
W20250110 02:51:10.544345 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 59900.3us
I20250110 02:51:10.544744 1214 consensus_queue.cc:1035] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:10.546550 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:10.547654 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7422cfed02fc4b50ab867fdf34769b58 (table default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]) successfully deleted
I20250110 02:51:10.549412 1315 raft_consensus.cc:2798] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:10.549577 1184 ts_tablet_manager.cc:1397] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.227s user 0.011s sys 0.003s
W20250110 02:51:10.549770 711 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.550029 1315 raft_consensus.cc:491] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:10.550122 1315 raft_consensus.cc:3054] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.552878 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb20d47ace4fd49f9dc4e526cab702. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.554440 709 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.554800 709 leader_election.cc:304] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:10.555620 1214 raft_consensus.cc:2743] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.561120 1184 raft_consensus.cc:357] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
W20250110 02:51:10.561642 996 consensus_peers.cc:487] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24efc81a5ac743809665b17f941c0652. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:10.570667 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_61492b773a714042b0532061e607c827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.571249 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61492b773a714042b0532061e607c827 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.572258 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_bcc7554985664d30b2fca81e32af5394" } new_table_name: "default.Table_0e304e93af93489cae36fca1cfd0fd8d" modify_external_catalogs: true
I20250110 02:51:10.572503 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 89f6797b1076468a93fb0b510c553bed
I20250110 02:51:10.573187 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_83ab4512090d4fd0b699700953b61a82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.573355 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1dfc062b5fcb42c3bce6dfaed54b3869
W20250110 02:51:10.573686 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83ab4512090d4fd0b699700953b61a82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.573751 517 catalog_manager.cc:6596] ReplaceTablet: tablet 1dfc062b5fcb42c3bce6dfaed54b3869 deleted and replaced by tablet 585320c0970c4097b195caaae33789b5
I20250110 02:51:10.573845 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 866513b2738844bba19bb91a7f4bfd81
I20250110 02:51:10.574069 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a0cd879bafe848c6bb8e40d099e6b735"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.574535 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6162127f127347419432d61cdac426bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.574829 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_018e09f0d16349738140c8eef71e9050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.575045 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6162127f127347419432d61cdac426bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:10.575287 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_018e09f0d16349738140c8eef71e9050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.576858 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 866513b2738844bba19bb91a7f4bfd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:10.574558 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0cd879bafe848c6bb8e40d099e6b735 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.577268 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.577535 519 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]). Expected version 1 got 0
I20250110 02:51:10.580299 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:10.580821 521 catalog_manager.cc:6596] ReplaceTablet: tablet c65704c9d486444ebe8e4ffd4580f395 deleted and replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b
I20250110 02:51:10.581038 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 0c2412420dc54258a6921ead194c1587 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.581130 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.581589 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_04563a8a322645f7b582860698d0e791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.582005 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04563a8a322645f7b582860698d0e791 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.585765 1180 ts_tablet_manager.cc:1918] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.585911 1180 log.cc:1198] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b85eea2ad540445dbd95a655135dc043
I20250110 02:51:10.586289 1180 ts_tablet_manager.cc:1939] T b85eea2ad540445dbd95a655135dc043 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.589089 1181 tablet_bootstrap.cc:492] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.589234 1181 ts_tablet_manager.cc:1397] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.273s user 0.007s sys 0.007s
I20250110 02:51:10.590090 1181 raft_consensus.cc:357] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.590281 1181 raft_consensus.cc:383] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.590373 1181 raft_consensus.cc:738] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.591092 1181 consensus_queue.cc:260] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.592198 1180 tablet_replica.cc:331] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.592379 1180 raft_consensus.cc:2238] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:10.592697 1180 pending_rounds.cc:62] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:10.592767 1180 pending_rounds.cc:69] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611715667251200 op_type: NO_OP noop_request { }
I20250110 02:51:10.592898 1180 raft_consensus.cc:2883] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:10.593001 1180 raft_consensus.cc:2267] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.592939 1181 ts_tablet_manager.cc:1428] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:10.594611 1181 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.599265 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.599334 1181 tablet_bootstrap.cc:654] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.599467 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.599702 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.599866 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 866513b2738844bba19bb91a7f4bfd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.600013 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.599718 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.601027 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.583535 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.583670 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.583966 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 866513b2738844bba19bb91a7f4bfd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.586593 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.601698 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.602015 1198 raft_consensus.cc:491] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.602154 1198 raft_consensus.cc:513] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.602197 1314 consensus_queue.cc:1035] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:10.602802 1198 leader_election.cc:290] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.602838 1178 ts_tablet_manager.cc:1918] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.602950 1178 log.cc:1198] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/9bce8f5888bb404696e46487176b5fe8
I20250110 02:51:10.603282 1178 ts_tablet_manager.cc:1939] T 9bce8f5888bb404696e46487176b5fe8 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.603251 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:10.604007 853 leader_election.cc:343] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:10.604305 1178 tablet_replica.cc:331] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.604557 1178 raft_consensus.cc:2238] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.604789 1178 raft_consensus.cc:2267] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.605410 1178 ts_tablet_manager.cc:1905] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.606163 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 9bce8f5888bb404696e46487176b5fe8 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]) successfully deleted
I20250110 02:51:10.610194 1180 ts_tablet_manager.cc:1905] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.610273 1188 tablet_bootstrap.cc:492] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.610435 1188 ts_tablet_manager.cc:1397] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.091s user 0.021s sys 0.030s
I20250110 02:51:10.611153 1188 raft_consensus.cc:357] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.611171 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a27f0327d1844b8a561af8b0bf4385b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:10.612049 853 leader_election.cc:343] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.612167 853 leader_election.cc:304] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.612545 1198 raft_consensus.cc:2743] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.612749 1188 raft_consensus.cc:383] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.613651 1181 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.613858 1181 ts_tablet_manager.cc:1397] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250110 02:51:10.614710 1181 raft_consensus.cc:357] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.615010 1181 raft_consensus.cc:383] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.615130 1181 raft_consensus.cc:738] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.615425 1181 consensus_queue.cc:260] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.616160 1181 ts_tablet_manager.cc:1428] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:10.616576 746 tablet_service.cc:1514] Processing DeleteTablet for tablet d58e92b9789e49f6bbfbefe3f8f3a0ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.616631 1181 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.616799 1315 raft_consensus.cc:513] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.617785 1315 leader_election.cc:290] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:10.618093 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d5f9bda55942ff856252bbb50b3c5b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:10.618296 1061 raft_consensus.cc:3054] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.618321 1181 tablet_bootstrap.cc:654] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.619232 1188 raft_consensus.cc:738] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.619517 1188 consensus_queue.cc:260] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.620077 1188 ts_tablet_manager.cc:1428] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:10.620327 1188 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.620627 1180 ts_tablet_manager.cc:1918] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.620760 1180 log.cc:1198] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/24efc81a5ac743809665b17f941c0652
I20250110 02:51:10.621201 1180 ts_tablet_manager.cc:1939] T 24efc81a5ac743809665b17f941c0652 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.621776 1061 raft_consensus.cc:2463] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:10.621917 1188 tablet_bootstrap.cc:654] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.622267 1180 tablet_replica.cc:331] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.622407 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 24efc81a5ac743809665b17f941c0652 (table default.Table_7192835167d842589c56b4b5432a247f [id=6b2d41d8c0a845be939307cb81f497e5]) successfully deleted
W20250110 02:51:10.622713 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4c9d791b1c504171ba66dec55c8fb0e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9d791b1c504171ba66dec55c8fb0e6 already in progress: creating tablet
I20250110 02:51:10.623181 711 leader_election.cc:304] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:10.623899 1315 raft_consensus.cc:2798] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:10.624284 1315 raft_consensus.cc:695] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:10.625631 1179 ts_tablet_manager.cc:1939] T 9bce8f5888bb404696e46487176b5fe8 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:10.626777 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 9bce8f5888bb404696e46487176b5fe8 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]) successfully deleted
I20250110 02:51:10.627094 1180 raft_consensus.cc:2238] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.627185 1179 tablet_replica.cc:331] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:10.627365 1180 raft_consensus.cc:2267] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.627460 1179 raft_consensus.cc:2238] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:10.627863 1179 raft_consensus.cc:2267] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.629930 1184 raft_consensus.cc:383] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.630203 1184 raft_consensus.cc:738] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.631094 1184 consensus_queue.cc:260] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.633378 1184 ts_tablet_manager.cc:1428] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.084s user 0.001s sys 0.001s
I20250110 02:51:10.634021 1184 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:10.635690 1184 tablet_bootstrap.cc:654] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.624675 1315 consensus_queue.cc:237] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.640444 1258 raft_consensus.cc:491] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.640573 1258 raft_consensus.cc:513] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.641191 1258 leader_election.cc:290] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.627979 1180 ts_tablet_manager.cc:1905] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.642311 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d5f9bda55942ff856252bbb50b3c5b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:10.642534 917 raft_consensus.cc:3054] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:10.643867 994 leader_election.cc:343] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 9bce8f5888bb404696e46487176b5fe8
W20250110 02:51:10.644601 996 leader_election.cc:343] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9bce8f5888bb404696e46487176b5fe8
I20250110 02:51:10.644721 996 leader_election.cc:304] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.645061 1258 raft_consensus.cc:2743] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.645634 1179 ts_tablet_manager.cc:1905] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.646886 917 raft_consensus.cc:2463] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:10.647805 1178 ts_tablet_manager.cc:1918] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.648010 1178 log.cc:1198] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d58e92b9789e49f6bbfbefe3f8f3a0ef
I20250110 02:51:10.648469 1178 ts_tablet_manager.cc:1939] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.648524 1180 ts_tablet_manager.cc:1918] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.648689 1180 log.cc:1198] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7422cfed02fc4b50ab867fdf34769b58
I20250110 02:51:10.648718 1184 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:10.648849 1184 ts_tablet_manager.cc:1397] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250110 02:51:10.651237 1181 tablet_bootstrap.cc:492] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.651376 1181 ts_tablet_manager.cc:1397] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.011s
I20250110 02:51:10.651677 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]) successfully deleted
I20250110 02:51:10.652072 1178 tablet_replica.cc:331] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.652156 1181 raft_consensus.cc:357] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.652334 1181 raft_consensus.cc:383] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.652429 1181 raft_consensus.cc:738] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.652545 1178 raft_consensus.cc:2238] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.652679 1181 consensus_queue.cc:260] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.653204 1181 ts_tablet_manager.cc:1428] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:10.653304 1178 raft_consensus.cc:2267] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.653462 1181 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.653810 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.654331 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1602ade49b7442d6b8e6f4b85d2e80a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.655220 1181 tablet_bootstrap.cc:654] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.656095 1178 ts_tablet_manager.cc:1905] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.656963 1188 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.657091 1188 ts_tablet_manager.cc:1397] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.011s
I20250110 02:51:10.657764 1188 raft_consensus.cc:357] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.657954 1188 raft_consensus.cc:383] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.658027 1188 raft_consensus.cc:738] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.658253 1188 consensus_queue.cc:260] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.659101 1188 ts_tablet_manager.cc:1428] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:10.659163 1180 ts_tablet_manager.cc:1939] T 7422cfed02fc4b50ab867fdf34769b58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.659371 1188 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.661082 1188 tablet_bootstrap.cc:654] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.661252 1180 tablet_replica.cc:331] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.661460 1180 raft_consensus.cc:2238] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.661422 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7422cfed02fc4b50ab867fdf34769b58 (table default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]) successfully deleted
I20250110 02:51:10.661602 1180 raft_consensus.cc:2267] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.662102 1180 ts_tablet_manager.cc:1905] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.664921 1184 raft_consensus.cc:357] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.665252 1184 raft_consensus.cc:383] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.665398 1184 raft_consensus.cc:738] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.665729 1184 consensus_queue.cc:260] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.666577 1184 ts_tablet_manager.cc:1428] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20250110 02:51:10.668247 1181 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.670197 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.670684 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1602ade49b7442d6b8e6f4b85d2e80a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.674474 1181 ts_tablet_manager.cc:1397] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20250110 02:51:10.675261 1178 ts_tablet_manager.cc:1918] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.675259 1181 raft_consensus.cc:357] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.675525 1178 log.cc:1198] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/10504e77a5c64ebf85be937d64ca9c80
I20250110 02:51:10.676061 1178 ts_tablet_manager.cc:1939] T 10504e77a5c64ebf85be937d64ca9c80 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.676339 1181 raft_consensus.cc:383] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.676458 1181 raft_consensus.cc:738] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.677469 1178 tablet_replica.cc:331] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.678100 1178 raft_consensus.cc:2238] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.678141 1181 consensus_queue.cc:260] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.678324 1178 raft_consensus.cc:2267] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.678756 1181 ts_tablet_manager.cc:1428] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:10.678815 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 10504e77a5c64ebf85be937d64ca9c80 (table default.Table_1a58fb7be3a34571bedcfbbff5db8ea7 [id=dd4008ea14ad40c68b3e71fac9e2be88]) successfully deleted
I20250110 02:51:10.679072 1181 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.680024 1178 ts_tablet_manager.cc:1905] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.680760 1181 tablet_bootstrap.cc:654] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:10.681097 709 consensus_peers.cc:487] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58e92b9789e49f6bbfbefe3f8f3a0ef. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:10.684722 890 tablet_service.cc:1467] Processing CreateTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a (DEFAULT_TABLE table=default.Table_1621908f975348db834b77c8e899eb67 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.685271 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f229f546a8d4a09b93f8b46c893d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.690929 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.691476 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1602ade49b7442d6b8e6f4b85d2e80a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.693491 1180 ts_tablet_manager.cc:1918] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.693621 1180 log.cc:1198] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/9bce8f5888bb404696e46487176b5fe8
I20250110 02:51:10.693971 1180 ts_tablet_manager.cc:1939] T 9bce8f5888bb404696e46487176b5fe8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.694900 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 9bce8f5888bb404696e46487176b5fe8 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]) successfully deleted
I20250110 02:51:10.696831 1178 ts_tablet_manager.cc:1918] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.696889 1181 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.697031 1178 log.cc:1198] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/80900124f93d43a0abf93175b16225a7
I20250110 02:51:10.697042 1181 ts_tablet_manager.cc:1397] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20250110 02:51:10.697449 1178 ts_tablet_manager.cc:1939] T 80900124f93d43a0abf93175b16225a7 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.697626 746 tablet_service.cc:1467] Processing CreateTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a (DEFAULT_TABLE table=default.Table_1621908f975348db834b77c8e899eb67 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.698056 1181 raft_consensus.cc:357] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.698288 1181 raft_consensus.cc:383] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.698381 1181 raft_consensus.cc:738] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.698573 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f229f546a8d4a09b93f8b46c893d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.698623 1181 consensus_queue.cc:260] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.699216 1181 ts_tablet_manager.cc:1428] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:10.699470 1181 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.699712 1180 tablet_replica.cc:331] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.699915 1180 raft_consensus.cc:2238] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.700098 1180 raft_consensus.cc:2267] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.701354 1181 tablet_bootstrap.cc:654] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.701326 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
W20250110 02:51:10.701978 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 70af6ec40c7a4442b91680d8125c2d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70af6ec40c7a4442b91680d8125c2d48 already in progress: creating tablet
I20250110 02:51:10.704507 1178 tablet_replica.cc:331] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.704798 1178 raft_consensus.cc:2238] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.704752 1198 raft_consensus.cc:491] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.704985 1198 raft_consensus.cc:513] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.705679 1198 leader_election.cc:290] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:10.705966 1178 raft_consensus.cc:2267] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.706221 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:10.706390 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866513b2738844bba19bb91a7f4bfd81" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:10.706622 776 raft_consensus.cc:2388] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:10.706907 1178 ts_tablet_manager.cc:1905] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.706984 853 leader_election.cc:304] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.707365 1198 raft_consensus.cc:3054] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.710338 1198 raft_consensus.cc:2743] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.710778 1315 consensus_queue.cc:1035] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:10.713405 1178 ts_tablet_manager.cc:1918] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.713555 1178 log.cc:1198] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c683bc3feaff44b1830cccfe839f677e
I20250110 02:51:10.713961 1178 ts_tablet_manager.cc:1939] T c683bc3feaff44b1830cccfe839f677e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.714839 1184 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:10.715226 1180 ts_tablet_manager.cc:1905] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.719195 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a (DEFAULT_TABLE table=default.Table_1621908f975348db834b77c8e899eb67 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.719547 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f229f546a8d4a09b93f8b46c893d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.725739 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 (DEFAULT_TABLE table=default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.726132 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abd85a070f74e9bb5dc4026935edcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.728210 1180 ts_tablet_manager.cc:1918] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.728312 1180 log.cc:1198] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c683bc3feaff44b1830cccfe839f677e
I20250110 02:51:10.728644 1180 ts_tablet_manager.cc:1939] T c683bc3feaff44b1830cccfe839f677e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.729856 1178 tablet_replica.cc:331] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.729954 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]) successfully deleted
I20250110 02:51:10.730022 1178 raft_consensus.cc:2238] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.730180 1178 raft_consensus.cc:2267] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:10.730499 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet ad9f920cde954f31acebc6d0b6965e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9f920cde954f31acebc6d0b6965e58 already in progress: creating tablet
I20250110 02:51:10.735862 1178 ts_tablet_manager.cc:1905] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.736299 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 (DEFAULT_TABLE table=default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.736771 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abd85a070f74e9bb5dc4026935edcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.740300 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 (DEFAULT_TABLE table=default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.740687 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abd85a070f74e9bb5dc4026935edcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.742291 1258 consensus_queue.cc:1035] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:10.744230 709 consensus_peers.cc:487] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b851ddf27fe44d19b9ca39d82e6352cd. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:10.747234 885 tablet_service.cc:1467] Processing CreateTablet for tablet c09235548b5d4e10a01cbe05d1c94841 (DEFAULT_TABLE table=default.Table_033d0d58d92e4a23ae7b1927e5db6163 [id=9e880e06a6d648b7bc44b651084a14a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.747648 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09235548b5d4e10a01cbe05d1c94841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.753755 917 raft_consensus.cc:1270] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Refusing update from remote peer 8899dd1aeb384e4ea4523f5fd225c888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:10.754360 754 tablet_service.cc:1467] Processing CreateTablet for tablet c09235548b5d4e10a01cbe05d1c94841 (DEFAULT_TABLE table=default.Table_033d0d58d92e4a23ae7b1927e5db6163 [id=9e880e06a6d648b7bc44b651084a14a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.754567 1258 consensus_queue.cc:1035] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:10.754861 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09235548b5d4e10a01cbe05d1c94841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.758567 1178 ts_tablet_manager.cc:1918] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.758702 1178 log.cc:1198] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/4c845c27a86d4e4f93c932313ebf648b
I20250110 02:51:10.759032 1178 ts_tablet_manager.cc:1939] T 4c845c27a86d4e4f93c932313ebf648b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.760082 1178 tablet_replica.cc:331] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.760270 1178 raft_consensus.cc:2238] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:10.760274 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet aa40d20288644b2a8d36ede967d16e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa40d20288644b2a8d36ede967d16e79 already in progress: creating tablet
I20250110 02:51:10.760397 1178 raft_consensus.cc:2267] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.760875 1178 ts_tablet_manager.cc:1905] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.764276 1184 tablet_bootstrap.cc:654] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.767958 893 tablet_service.cc:1467] Processing CreateTablet for tablet 361c332d66984cf7ab84d74e913a5bde (DEFAULT_TABLE table=default.Table_0595ee2f9d4442cca8e712debfae068e [id=0692620395c14593803592e6aade6af3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.768302 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361c332d66984cf7ab84d74e913a5bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.769932 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 80900124f93d43a0abf93175b16225a7 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.771023 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_17a3ad3c46d2416db2b8bd0f66e39703" } new_table_name: "default.Table_73580b26ac6a47ab81519905e8adfb2f" modify_external_catalogs: true
I20250110 02:51:10.771919 525 catalog_manager.cc:5526] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:10.772289 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]). Expected version 1 got 0
I20250110 02:51:10.773022 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 4e2c01af512f460daa98b19444f72172
I20250110 02:51:10.773736 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1a56c4b5a83345798b5d0f524ba0560d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.774158 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a56c4b5a83345798b5d0f524ba0560d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.774413 1188 tablet_bootstrap.cc:492] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:10.775848 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_59d402904b724493ba7a5f2a0710b283" } new_table_name: "default.Table_bc03ee1fa76e458c8029c902c813dcb2" modify_external_catalogs: true
I20250110 02:51:10.776895 1179 ts_tablet_manager.cc:1918] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.777027 1179 log.cc:1198] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c683bc3feaff44b1830cccfe839f677e
I20250110 02:51:10.777455 1179 ts_tablet_manager.cc:1939] T c683bc3feaff44b1830cccfe839f677e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:10.777472 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7489371158674f9abe9b0cafbd4f1157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:10.777957 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7489371158674f9abe9b0cafbd4f1157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.778604 1179 tablet_replica.cc:331] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:10.778836 1179 raft_consensus.cc:2238] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.779001 1179 raft_consensus.cc:2267] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.779428 1179 ts_tablet_manager.cc:1905] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.781587 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f86542fc90134a57b994da74f3e5b461" } new_table_name: "default.Table_dce75d4fa88e4bf59fe18cf4725d953f" modify_external_catalogs: true
I20250110 02:51:10.785135 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c683bc3feaff44b1830cccfe839f677e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.786234 1180 tablet_replica.cc:331] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.789105 1180 raft_consensus.cc:2238] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.789290 1180 raft_consensus.cc:2267] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.789784 1180 ts_tablet_manager.cc:1905] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.789875 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]) successfully deleted
I20250110 02:51:10.791525 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b397397d667841eda2b5a948ca4a5015"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:10.791729 750 tablet_service.cc:1467] Processing CreateTablet for tablet 2e10e7654cec495bb532fb00e73c020d (DEFAULT_TABLE table=default.Table_a037268bb40540c7b7495367b363d30b [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:10.792096 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b397397d667841eda2b5a948ca4a5015 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.792186 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:10.793370 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b74812cb81c44fcb8786cc5dae328e3 consensus state has no leader
I20250110 02:51:10.793846 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 886ab6d12cf545b9b700959e18396c50 with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d4c4631994a5792200ba4d4c906f0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.794072 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.794441 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 361c332d66984cf7ab84d74e913a5bde (DEFAULT_TABLE table=default.Table_0595ee2f9d4442cca8e712debfae068e [id=0692620395c14593803592e6aade6af3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.795524 1180 ts_tablet_manager.cc:1918] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.795038 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.795256 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 2e10e7654cec495bb532fb00e73c020d (DEFAULT_TABLE table=default.Table_a037268bb40540c7b7495367b363d30b [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.796396 754 tablet_service.cc:1467] Processing CreateTablet for tablet 361c332d66984cf7ab84d74e913a5bde (DEFAULT_TABLE table=default.Table_0595ee2f9d4442cca8e712debfae068e [id=0692620395c14593803592e6aade6af3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.797235 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361c332d66984cf7ab84d74e913a5bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.797416 1033 tablet_service.cc:1467] Processing CreateTablet for tablet c09235548b5d4e10a01cbe05d1c94841 (DEFAULT_TABLE table=default.Table_033d0d58d92e4a23ae7b1927e5db6163 [id=9e880e06a6d648b7bc44b651084a14a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.797155 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:10.800390 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 70af6ec40c7a4442b91680d8125c2d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70af6ec40c7a4442b91680d8125c2d48 already in progress: creating tablet
W20250110 02:51:10.800673 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet ad9f920cde954f31acebc6d0b6965e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9f920cde954f31acebc6d0b6965e58 already in progress: creating tablet
I20250110 02:51:10.800961 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c09235548b5d4e10a01cbe05d1c94841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.801152 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e10e7654cec495bb532fb00e73c020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.801124 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c683bc3feaff44b1830cccfe839f677e (table default.Table_d24028774b004dbfabe940b74a12bd16 [id=82b5379bd1dc4c09a2aac28e5583e04d]) successfully deleted
I20250110 02:51:10.801358 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8940c201aa104b0b8bc8d866859fe8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.802120 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 0c2412420dc54258a6921ead194c1587
I20250110 02:51:10.802673 521 catalog_manager.cc:6596] ReplaceTablet: tablet 0c2412420dc54258a6921ead194c1587 deleted and replaced by tablet f14f06ba7d0f44418693e36f6bc6d888
I20250110 02:51:10.803031 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b439c40f07fd4b48b1735826b7eedbf0" } new_table_name: "default.Table_ed90cac8f4544e85bab3425999a77496" modify_external_catalogs: true
I20250110 02:51:10.811821 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.815049 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.818535 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1a58fb7be3a34571bedcfbbff5db8ea7" } modify_external_catalogs: true
I20250110 02:51:10.818727 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1a58fb7be3a34571bedcfbbff5db8ea7" } modify_external_catalogs: true
I20250110 02:51:10.818802 1180 log.cc:1198] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/80900124f93d43a0abf93175b16225a7
I20250110 02:51:10.819183 1180 ts_tablet_manager.cc:1939] T 80900124f93d43a0abf93175b16225a7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.821491 1188 ts_tablet_manager.cc:1397] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.027s
I20250110 02:51:10.822067 889 tablet_service.cc:1467] Processing CreateTablet for tablet 2e10e7654cec495bb532fb00e73c020d (DEFAULT_TABLE table=default.Table_a037268bb40540c7b7495367b363d30b [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.823966 896 tablet_service.cc:1467] Processing CreateTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.824666 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa76357abae4a95b8fb3af47288e43a (DEFAULT_TABLE table=default.Table_f86542fc90134a57b994da74f3e5b461 [id=7c8bb5b8a91a4ed9a81403e2e6196406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.828505 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa76357abae4a95b8fb3af47288e43a (DEFAULT_TABLE table=default.Table_f86542fc90134a57b994da74f3e5b461 [id=7c8bb5b8a91a4ed9a81403e2e6196406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.828946 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa76357abae4a95b8fb3af47288e43a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.831894 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da8f247c862e4cb68452e230857ee6b0 has no consensus state
I20250110 02:51:10.832378 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a788c9c301f14b34b873006c6715a243" } modify_external_catalogs: true
I20250110 02:51:10.832612 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a788c9c301f14b34b873006c6715a243" } modify_external_catalogs: true
W20250110 02:51:10.833387 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e72ed2d31a8467f8c092b48d593c1cc has no consensus state
I20250110 02:51:10.833681 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3e5d4c967ed04133a245f4185b257710" } modify_external_catalogs: true
I20250110 02:51:10.833829 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3e5d4c967ed04133a245f4185b257710" } modify_external_catalogs: true
I20250110 02:51:10.834383 517 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c4ddce8ad8234c6cb20e6015e489a340
I20250110 02:51:10.834697 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5" } modify_external_catalogs: true
I20250110 02:51:10.834858 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5" } modify_external_catalogs: true
W20250110 02:51:10.835461 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fa76357abae4a95b8fb3af47288e43a consensus state has no leader
W20250110 02:51:10.842324 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe86be26d1b544c08b231f20085d3159 consensus state has no leader
I20250110 02:51:10.843436 1180 tablet_replica.cc:331] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.844250 1180 raft_consensus.cc:2238] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.845077 1180 raft_consensus.cc:2267] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.846124 1180 ts_tablet_manager.cc:1905] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.846155 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
I20250110 02:51:10.847146 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.847611 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.848664 745 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (DEFAULT_TABLE table=default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.850242 1034 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (DEFAULT_TABLE table=default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.850740 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.851576 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.855720 1354 tablet.cc:1721] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:10.856798 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa76357abae4a95b8fb3af47288e43a (DEFAULT_TABLE table=default.Table_f86542fc90134a57b994da74f3e5b461 [id=7c8bb5b8a91a4ed9a81403e2e6196406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.857255 892 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (DEFAULT_TABLE table=default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.857560 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e10e7654cec495bb532fb00e73c020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.857720 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.859022 1184 tablet_bootstrap.cc:492] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:10.859180 1184 ts_tablet_manager.cc:1397] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.144s user 0.002s sys 0.003s
I20250110 02:51:10.859472 749 tablet_service.cc:1467] Processing CreateTablet for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (DEFAULT_TABLE table=default.Table_bfd353801c504f1aab972805a7301c22 [id=1afd8993888840da819f652f440b1b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.859921 1178 ts_tablet_manager.cc:1918] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.859978 1184 raft_consensus.cc:357] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.860162 1184 raft_consensus.cc:383] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.860499 1184 raft_consensus.cc:738] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.860828 1258 raft_consensus.cc:491] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.860774 1184 consensus_queue.cc:260] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.860960 1258 raft_consensus.cc:513] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.861652 1258 leader_election.cc:290] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.862493 1184 ts_tablet_manager.cc:1428] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:10.862831 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:10.863157 996 leader_election.cc:343] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 886ab6d12cf545b9b700959e18396c50
I20250110 02:51:10.863211 1184 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:10.866159 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 423c820b81b845719d59a376c18e175a
I20250110 02:51:10.866483 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:10.866922 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.868074 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 05d5f9bda55942ff856252bbb50b3c5b
I20250110 02:51:10.868665 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.868996 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.867641 1188 raft_consensus.cc:357] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.869489 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.869691 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.869932 1178 log.cc:1198] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/89984fa8f39641a39969f4e3392ae8b7
I20250110 02:51:10.870023 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.870958 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.871315 753 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.871672 1178 ts_tablet_manager.cc:1939] T 89984fa8f39641a39969f4e3392ae8b7 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.872249 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f6f4f221e6445db0bb569f372cfaa0 with delete_type TABLET_DATA_DELETED (replaced by tablet 825b4d370dab4b908ddde0f1dddf466f) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.873466 1035 tablet_service.cc:1467] Processing CreateTablet for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (DEFAULT_TABLE table=default.Table_bfd353801c504f1aab972805a7301c22 [id=1afd8993888840da819f652f440b1b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.874552 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.874724 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.874771 1181 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:10.874886 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 05d5f9bda55942ff856252bbb50b3c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.874905 1181 ts_tablet_manager.cc:1397] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.175s user 0.035s sys 0.051s
I20250110 02:51:10.877126 1181 raft_consensus.cc:357] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.877342 1181 raft_consensus.cc:383] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.877404 1181 raft_consensus.cc:738] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.877637 1181 consensus_queue.cc:260] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:10.879024 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.879046 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 89984fa8f39641a39969f4e3392ae8b7 (table default.Table_46bcf45f1fce419c882b8c239e680ee9 [id=1da07435b54849318beda7e8518f0bf6]) successfully deleted
I20250110 02:51:10.879231 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec427cc101bd49f6b9d1743e3eeebbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.879258 891 tablet_service.cc:1467] Processing CreateTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.879563 1181 ts_tablet_manager.cc:1428] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:10.879761 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa76357abae4a95b8fb3af47288e43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.879427 753 tablet_service.cc:1467] Processing CreateTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.878768 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a27f0327d1844b8a561af8b0bf4385b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.882028 1184 tablet_bootstrap.cc:654] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.882761 746 tablet_service.cc:1467] Processing CreateTablet for tablet 344184e1c98d422095cb254f5b2bc792 (DEFAULT_TABLE table=default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.885464 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 05d5f9bda55942ff856252bbb50b3c5b from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.886317 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "886ab6d12cf545b9b700959e18396c50" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.887372 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8940c201aa104b0b8bc8d866859fe8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.888494 1188 raft_consensus.cc:383] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:10.888576 885 tablet_service.cc:1467] Processing CreateTablet for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (DEFAULT_TABLE table=default.Table_bfd353801c504f1aab972805a7301c22 [id=1afd8993888840da819f652f440b1b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.887939 1302 mvcc.cc:204] Tried to move back new op lower bound from 7112611720197332992 to 7112611716747055104. Current Snapshot: MvccSnapshot[applied={T|T < 7112611720197332992}]
I20250110 02:51:10.889731 1178 tablet_replica.cc:331] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:10.890966 1181 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:10.894819 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019c0d60da5548bb86c93ccd89bf94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.897275 1188 raft_consensus.cc:738] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:10.898088 1178 raft_consensus.cc:2238] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.898103 886 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d4c4631994a5792200ba4d4c906f0 (DEFAULT_TABLE table=default.Table_8eb6afa433004d3aaac88418b07a91f3 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.902081 1178 raft_consensus.cc:2267] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.903311 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 89f6797b1076468a93fb0b510c553bed (DEFAULT_TABLE table=default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.904001 889 tablet_service.cc:1467] Processing CreateTablet for tablet 344184e1c98d422095cb254f5b2bc792 (DEFAULT_TABLE table=default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.904790 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f6797b1076468a93fb0b510c553bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.905237 1178 ts_tablet_manager.cc:1905] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.906075 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 0b74812cb81c44fcb8786cc5dae328e3 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:10.906821 994 leader_election.cc:343] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.905081 1188 consensus_queue.cc:260] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:10.909824 1030 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d4c4631994a5792200ba4d4c906f0 (DEFAULT_TABLE table=default.Table_8eb6afa433004d3aaac88418b07a91f3 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.910194 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d4c4631994a5792200ba4d4c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.912073 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 344184e1c98d422095cb254f5b2bc792 (DEFAULT_TABLE table=default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.912437 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344184e1c98d422095cb254f5b2bc792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.915269 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 0b74812cb81c44fcb8786cc5dae328e3
I20250110 02:51:10.915663 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d4c4631994a5792200ba4d4c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.915732 520 catalog_manager.cc:6596] ReplaceTablet: tablet 0b74812cb81c44fcb8786cc5dae328e3 deleted and replaced by tablet e8b2297377bc4e5095826e30523eede7
I20250110 02:51:10.916052 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff (DEFAULT_TABLE table=default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.916178 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:10.916420 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019c0d60da5548bb86c93ccd89bf94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.916828 994 leader_election.cc:304] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.917304 525 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 67adef4c3f4041e1ab480100ca762f62 (table default.Table_15356bd1a2bd4846be42885b635b99ad [id=9c8eec49718f4b089953cb1ab7b7a57a]). Expected version 2 got 0
I20250110 02:51:10.917362 1352 raft_consensus.cc:2743] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:10.918901 1356 tablet.cc:1721] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:10.919840 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:10.921788 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (DEFAULT_TABLE table=default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.918398 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344184e1c98d422095cb254f5b2bc792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.922993 1180 ts_tablet_manager.cc:1918] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 02:51:10.923345 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250110 02:51:10.923768 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3934e6611d4e4b94ba5185f33c3d05f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:10.924382 510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd -> Peer 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025): Couldn't send request to peer 5d8d40556dc348e8aa8255d116251a10. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250110 02:51:10.925539 1188 ts_tablet_manager.cc:1428] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.102s user 0.000s sys 0.002s
I20250110 02:51:10.926110 1188 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:10.928069 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (DEFAULT_TABLE table=default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.928421 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3934e6611d4e4b94ba5185f33c3d05f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.929303 1188 tablet_bootstrap.cc:654] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:10.929622 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec427cc101bd49f6b9d1743e3eeebbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.932312 1180 log.cc:1198] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d58e92b9789e49f6bbfbefe3f8f3a0ef
I20250110 02:51:10.934825 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e10e7654cec495bb532fb00e73c020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.936144 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8940c201aa104b0b8bc8d866859fe8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.936664 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (DEFAULT_TABLE table=default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.937150 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3934e6611d4e4b94ba5185f33c3d05f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.937948 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361c332d66984cf7ab84d74e913a5bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.938970 1181 tablet_bootstrap.cc:654] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:10.939487 711 consensus_peers.cc:487] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:10.941519 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:10.941857 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 423c820b81b845719d59a376c18e175a from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:10.942315 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa76357abae4a95b8fb3af47288e43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.942296 1352 raft_consensus.cc:491] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:10.942338 755 tablet_service.cc:1467] Processing CreateTablet for tablet 89f6797b1076468a93fb0b510c553bed (DEFAULT_TABLE table=default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.943089 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019c0d60da5548bb86c93ccd89bf94ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.943282 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.943507 752 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d4c4631994a5792200ba4d4c906f0 (DEFAULT_TABLE table=default.Table_8eb6afa433004d3aaac88418b07a91f3 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:10.943512 1352 raft_consensus.cc:513] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.944569 1352 leader_election.cc:290] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.944741 1180 ts_tablet_manager.cc:1939] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.945748 1180 tablet_replica.cc:331] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:10.946146 711 consensus_peers.cc:487] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89984fa8f39641a39969f4e3392ae8b7. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:10.947892 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec427cc101bd49f6b9d1743e3eeebbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.948014 1180 raft_consensus.cc:2238] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.948199 1180 raft_consensus.cc:2267] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.948321 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]) successfully deleted
W20250110 02:51:10.948582 512 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 286aee6074e14c27b17bc79a78906620 because the tablet was not found. No further retry: Not found: Tablet not found: 286aee6074e14c27b17bc79a78906620
I20250110 02:51:10.949972 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:10.950731 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:10.951329 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344184e1c98d422095cb254f5b2bc792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.951419 918 raft_consensus.cc:2463] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:10.956661 1179 ts_tablet_manager.cc:1918] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.956779 1179 log.cc:1198] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/286aee6074e14c27b17bc79a78906620
I20250110 02:51:10.957077 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f6797b1076468a93fb0b510c553bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.963996 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d4c4631994a5792200ba4d4c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:10.965389 1179 ts_tablet_manager.cc:1939] T 286aee6074e14c27b17bc79a78906620 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:10.967553 996 leader_election.cc:343] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.968025 994 leader_election.cc:304] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.968452 1352 raft_consensus.cc:2798] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:10.968530 1352 raft_consensus.cc:491] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:10.968607 1352 raft_consensus.cc:3054] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:10.972203 1180 ts_tablet_manager.cc:1905] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.970898 1352 raft_consensus.cc:513] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.977260 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:10.975831 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:10.978365 918 raft_consensus.cc:3054] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:10.978889 996 leader_election.cc:343] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:10.979482 1352 leader_election.cc:290] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:10.981192 918 raft_consensus.cc:2463] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:10.981652 994 leader_election.cc:304] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:10.975322 1178 ts_tablet_manager.cc:1918] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:10.982316 1178 log.cc:1198] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/886ab6d12cf545b9b700959e18396c50
I20250110 02:51:10.982919 1178 ts_tablet_manager.cc:1939] T 886ab6d12cf545b9b700959e18396c50 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:10.984395 1352 raft_consensus.cc:2798] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:10.985103 1352 raft_consensus.cc:695] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:10.985515 1352 consensus_queue.cc:237] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:10.987807 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 886ab6d12cf545b9b700959e18396c50 (table default.Table_8eb6afa433004d3aaac88418b07a91f3 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]) successfully deleted
I20250110 02:51:10.989252 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 286aee6074e14c27b17bc79a78906620 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
I20250110 02:51:10.991922 1180 ts_tablet_manager.cc:1918] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:10.993146 1180 log.cc:1198] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:10.993746 1180 ts_tablet_manager.cc:1939] T b851ddf27fe44d19b9ca39d82e6352cd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:10.999222 1180 tablet_replica.cc:331] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:10.999666 1180 raft_consensus.cc:2238] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:10.984824 890 tablet_service.cc:1467] Processing CreateTablet for tablet 89f6797b1076468a93fb0b510c553bed (DEFAULT_TABLE table=default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.000217 1180 pending_rounds.cc:62] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:11.000281 1180 pending_rounds.cc:69] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611717304446976 op_type: NO_OP noop_request { }
I20250110 02:51:11.000757 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f6797b1076468a93fb0b510c553bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.000838 1180 raft_consensus.cc:2883] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.000933 1179 tablet_replica.cc:331] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.001158 1180 raft_consensus.cc:2267] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.002209 1180 ts_tablet_manager.cc:1905] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.001385 1178 tablet_replica.cc:331] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:11.006667 1178 raft_consensus.cc:2238] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:10.990079 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.007330 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.990641 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5b2da56ee56b45bebccadc94ae4578a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:11.008412 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b851ddf27fe44d19b9ca39d82e6352cd (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
W20250110 02:51:11.008769 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b2da56ee56b45bebccadc94ae4578a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.991004 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_660c31e76a8c4a8ea394d251dd4718f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.009927 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_660c31e76a8c4a8ea394d251dd4718f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:10.993764 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_eea9dbdd7e284980bec035a0b1dc0c4f" } modify_external_catalogs: true
I20250110 02:51:11.011092 1179 raft_consensus.cc:2238] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.011111 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_eea9dbdd7e284980bec035a0b1dc0c4f" } modify_external_catalogs: true
I20250110 02:51:11.011376 1179 pending_rounds.cc:62] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:10.994349 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5d9b00106d6148598a57409f1df9ad37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.011672 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d9b00106d6148598a57409f1df9ad37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.012044 1179 pending_rounds.cc:69] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611716683706368 op_type: NO_OP noop_request { }
I20250110 02:51:11.012171 1179 raft_consensus.cc:2883] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.012488 1179 raft_consensus.cc:2267] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:10.999619 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_bfd353801c504f1aab972805a7301c22" } new_table_name: "default.Table_ee8b07a158764473a98e0582c12d2075" modify_external_catalogs: true
I20250110 02:51:11.012995 1179 ts_tablet_manager.cc:1905] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.013670 1180 ts_tablet_manager.cc:1918] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.013795 1180 log.cc:1198] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/10504e77a5c64ebf85be937d64ca9c80
I20250110 02:51:11.014212 1180 ts_tablet_manager.cc:1939] T 10504e77a5c64ebf85be937d64ca9c80 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:11.023315 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_28491d38435d448b9ecbb83b5efb94d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.023831 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28491d38435d448b9ecbb83b5efb94d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:11.031152 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet aa40d20288644b2a8d36ede967d16e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa40d20288644b2a8d36ede967d16e79 already in progress: creating tablet
I20250110 02:51:11.031682 1180 tablet_replica.cc:331] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:11.032094 1180 raft_consensus.cc:2238] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.032507 1180 raft_consensus.cc:2267] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.033185 1180 ts_tablet_manager.cc:1905] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:10.998764 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_50adace768fa44c698737e7843f5ac75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.033916 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50adace768fa44c698737e7843f5ac75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.036716 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_18721d8cd9034ade8df5d1035133a6a1" } modify_external_catalogs: true
I20250110 02:51:11.036953 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_18721d8cd9034ade8df5d1035133a6a1" } modify_external_catalogs: true
I20250110 02:51:11.037961 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_accf846f8cb6482db39436051e3fdd6f" } new_table_name: "default.Table_420beb896535455a87ee0a8c6a6049d4" modify_external_catalogs: true
I20250110 02:51:10.999938 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_cf93380c055c4e4c8c9fe4e359e5b874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.041659 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf93380c055c4e4c8c9fe4e359e5b874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.042088 1180 ts_tablet_manager.cc:1918] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.046181 1178 raft_consensus.cc:2267] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.049674 1180 log.cc:1198] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/4c845c27a86d4e4f93c932313ebf648b
I20250110 02:51:11.050254 1180 ts_tablet_manager.cc:1939] T 4c845c27a86d4e4f93c932313ebf648b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:11.051594 1178 ts_tablet_manager.cc:1905] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.052711 1179 ts_tablet_manager.cc:1918] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.052841 1179 log.cc:1198] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/89984fa8f39641a39969f4e3392ae8b7
I20250110 02:51:11.053202 1179 ts_tablet_manager.cc:1939] T 89984fa8f39641a39969f4e3392ae8b7 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.054186 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0e304e93af93489cae36fca1cfd0fd8d" } new_table_name: "default.Table_ee79c2aff75d4520b5ba281bc932d9d7" modify_external_catalogs: true
I20250110 02:51:11.054761 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_15356bd1a2bd4846be42885b635b99ad" } new_table_name: "default.Table_72de93e5a855471e85416b410edf348d" modify_external_catalogs: true
I20250110 02:51:11.056078 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (DEFAULT_TABLE table=default.Table_1a56c4b5a83345798b5d0f524ba0560d [id=f4cb430290954cba98bf9c6c2e45cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.056456 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60486a3dbc5a48eaa26735b84ae3fd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.057379 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a4b99ef64b874b609f835a3e58454edd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.058037 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4b99ef64b874b609f835a3e58454edd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.061017 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d8da12d0416f480794b1d69b6c62f386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.061551 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8da12d0416f480794b1d69b6c62f386 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.062849 746 tablet_service.cc:1467] Processing CreateTablet for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (DEFAULT_TABLE table=default.Table_1a56c4b5a83345798b5d0f524ba0560d [id=f4cb430290954cba98bf9c6c2e45cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.063047 1040 tablet_service.cc:1467] Processing CreateTablet for tablet cf711522866d46439d7ee69aa63d7f39 (DEFAULT_TABLE table=default.Table_b397397d667841eda2b5a948ca4a5015 [id=b0585ee264c44482941397f09c2eebb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.063202 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60486a3dbc5a48eaa26735b84ae3fd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.063334 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf711522866d46439d7ee69aa63d7f39. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.064062 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec427cc101bd49f6b9d1743e3eeebbd1 consensus state has no leader
I20250110 02:51:11.064460 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.064965 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.068089 752 tablet_service.cc:1467] Processing CreateTablet for tablet cf711522866d46439d7ee69aa63d7f39 (DEFAULT_TABLE table=default.Table_b397397d667841eda2b5a948ca4a5015 [id=b0585ee264c44482941397f09c2eebb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.068434 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf711522866d46439d7ee69aa63d7f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.071965 1179 tablet_replica.cc:331] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.072160 1179 raft_consensus.cc:2238] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.072685 1179 raft_consensus.cc:2267] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.074416 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 3352f8378b9846a3854086293f2d4e15 (DEFAULT_TABLE table=default.Table_04563a8a322645f7b582860698d0e791 [id=02e6300bb79544e99ddcee13edb258ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.074712 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3352f8378b9846a3854086293f2d4e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.074705 1179 ts_tablet_manager.cc:1905] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.076310 1181 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:11.076467 1181 ts_tablet_manager.cc:1397] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.195s user 0.013s sys 0.013s
I20250110 02:51:11.068918 886 tablet_service.cc:1467] Processing CreateTablet for tablet b6757b7ecc514489a0fa7e1ff18206bd (DEFAULT_TABLE table=default.Table_018e09f0d16349738140c8eef71e9050 [id=015845fa33aa4443b36402d79b4c94bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.079156 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0d22fd75fb474ebf07b624499a6f7b (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.079622 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0d22fd75fb474ebf07b624499a6f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.077183 1181 raft_consensus.cc:357] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.081430 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b6757b7ecc514489a0fa7e1ff18206bd (DEFAULT_TABLE table=default.Table_018e09f0d16349738140c8eef71e9050 [id=015845fa33aa4443b36402d79b4c94bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.081516 1181 raft_consensus.cc:383] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.081596 1181 raft_consensus.cc:738] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.081748 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6757b7ecc514489a0fa7e1ff18206bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.081808 1181 consensus_queue.cc:260] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.082285 1181 ts_tablet_manager.cc:1428] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:11.082767 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 (DEFAULT_TABLE table=default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.083082 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af3d61366b2432db53b18f2b7bb49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.085796 1181 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:11.086887 884 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4422eca5604d149e73c2a669d07af0 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.087379 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4422eca5604d149e73c2a669d07af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.088075 746 tablet_service.cc:1467] Processing CreateTablet for tablet b6757b7ecc514489a0fa7e1ff18206bd (DEFAULT_TABLE table=default.Table_018e09f0d16349738140c8eef71e9050 [id=015845fa33aa4443b36402d79b4c94bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.088418 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6757b7ecc514489a0fa7e1ff18206bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.094700 1034 tablet_service.cc:1467] Processing CreateTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf (DEFAULT_TABLE table=default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.095676 897 tablet_service.cc:1467] Processing CreateTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa (DEFAULT_TABLE table=default.Table_a0cd879bafe848c6bb8e40d099e6b735 [id=488d922a6b604812af65ff6b4d9c6606]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.095764 881 tablet_service.cc:1467] Processing CreateTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 (DEFAULT_TABLE table=default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.096714 753 tablet_service.cc:1467] Processing CreateTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf (DEFAULT_TABLE table=default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.096944 1033 tablet_service.cc:1467] Processing CreateTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (DEFAULT_TABLE table=default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.097110 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e744cdc1fcb94d46b48b684763ecf1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.098099 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 585320c0970c4097b195caaae33789b5 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.098753 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 46a4745356e3423f9d4892d242da1c48 (DEFAULT_TABLE table=default.Table_215c0fc0ef884696ad73db0829ca7e67 [id=3aad422d584f40df9d9241586adbc66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.099107 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad0d561a8ef498ba13bc813d4cd82fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.099871 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14f06ba7d0f44418693e36f6bc6d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.100416 747 tablet_service.cc:1467] Processing CreateTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (DEFAULT_TABLE table=default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.100715 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.105741 749 tablet_service.cc:1467] Processing CreateTablet for tablet 585320c0970c4097b195caaae33789b5 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.106057 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585320c0970c4097b195caaae33789b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.109119 711 consensus_peers.cc:487] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80900124f93d43a0abf93175b16225a7. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:11.109382 750 tablet_service.cc:1467] Processing CreateTablet for tablet 46a4745356e3423f9d4892d242da1c48 (DEFAULT_TABLE table=default.Table_215c0fc0ef884696ad73db0829ca7e67 [id=3aad422d584f40df9d9241586adbc66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.109707 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a4745356e3423f9d4892d242da1c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.109707 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b (DEFAULT_TABLE table=default.Table_87db4b399a114ebd9397992a26db0aba [id=62df87bfc41d477fb6044eff0ed5c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.110708 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 89984fa8f39641a39969f4e3392ae8b7 (table default.Table_46bcf45f1fce419c882b8c239e680ee9 [id=1da07435b54849318beda7e8518f0bf6]) successfully deleted
I20250110 02:51:11.111694 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 (DEFAULT_TABLE table=default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.111833 1198 raft_consensus.cc:491] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.112006 1198 raft_consensus.cc:513] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.112520 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4422eca5604d149e73c2a669d07af0 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.112778 1198 leader_election.cc:290] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:11.113189 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4422eca5604d149e73c2a669d07af0 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.113507 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4422eca5604d149e73c2a669d07af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.113709 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3352f8378b9846a3854086293f2d4e15 (DEFAULT_TABLE table=default.Table_04563a8a322645f7b582860698d0e791 [id=02e6300bb79544e99ddcee13edb258ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.113926 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:11.114050 743 tablet_service.cc:1467] Processing CreateTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 (DEFAULT_TABLE table=default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.114145 776 raft_consensus.cc:2463] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:11.114707 754 tablet_service.cc:1467] Processing CreateTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa (DEFAULT_TABLE table=default.Table_a0cd879bafe848c6bb8e40d099e6b735 [id=488d922a6b604812af65ff6b4d9c6606]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.115195 853 leader_election.cc:304] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.116154 1198 raft_consensus.cc:2798] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:11.116246 1198 raft_consensus.cc:491] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:11.116330 1198 raft_consensus.cc:3054] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.122077 1198 raft_consensus.cc:513] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.090889 1179 ts_tablet_manager.cc:1918] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.130797 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc (DEFAULT_TABLE table=default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.130964 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0b91f351e02d406f8675eb44c4c04ced (DEFAULT_TABLE table=default.Table_7489371158674f9abe9b0cafbd4f1157 [id=dfb29df1e3c54611abc07a3326df3215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.131035 879 tablet_service.cc:1467] Processing CreateTablet for tablet da8f247c862e4cb68452e230857ee6b0 (DEFAULT_TABLE table=default.Table_bc03ee1fa76e458c8029c902c813dcb2 [id=e2a6764b09814b9cb1367abb1a4f5ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.131336 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e72ed2d31a8467f8c092b48d593c1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.131562 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b91f351e02d406f8675eb44c4c04ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.132966 1198 leader_election.cc:290] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:11.069734 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3352f8378b9846a3854086293f2d4e15 (DEFAULT_TABLE table=default.Table_04563a8a322645f7b582860698d0e791 [id=02e6300bb79544e99ddcee13edb258ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.134613 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6757b7ecc514489a0fa7e1ff18206bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.137068 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb5412b78104d8085878c58b191ab27 (DEFAULT_TABLE table=default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.137434 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0d22fd75fb474ebf07b624499a6f7b (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.070320 891 tablet_service.cc:1467] Processing CreateTablet for tablet cf711522866d46439d7ee69aa63d7f39 (DEFAULT_TABLE table=default.Table_b397397d667841eda2b5a948ca4a5015 [id=b0585ee264c44482941397f09c2eebb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.070736 893 tablet_service.cc:1467] Processing CreateTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf (DEFAULT_TABLE table=default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.139648 897 tablet_service.cc:1467] Processing CreateTablet for tablet e8b2297377bc4e5095826e30523eede7 (DEFAULT_TABLE table=default.Table_0e304e93af93489cae36fca1cfd0fd8d [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.140195 881 tablet_service.cc:1467] Processing CreateTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 (DEFAULT_TABLE table=default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.140535 895 tablet_service.cc:1467] Processing CreateTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.071094 889 tablet_service.cc:1467] Processing CreateTablet for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (DEFAULT_TABLE table=default.Table_1a56c4b5a83345798b5d0f524ba0560d [id=f4cb430290954cba98bf9c6c2e45cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.072223 896 tablet_service.cc:1467] Processing CreateTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (DEFAULT_TABLE table=default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.073951 892 tablet_service.cc:1467] Processing CreateTablet for tablet 585320c0970c4097b195caaae33789b5 (DEFAULT_TABLE table=default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.074309 898 tablet_service.cc:1467] Processing CreateTablet for tablet 46a4745356e3423f9d4892d242da1c48 (DEFAULT_TABLE table=default.Table_215c0fc0ef884696ad73db0829ca7e67 [id=3aad422d584f40df9d9241586adbc66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.075865 888 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b (DEFAULT_TABLE table=default.Table_87db4b399a114ebd9397992a26db0aba [id=62df87bfc41d477fb6044eff0ed5c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:11.152724 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c0d22fd75fb474ebf07b624499a6f7b consensus state has no leader
I20250110 02:51:11.153219 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8da521fca5664959b1c0801b30199a6f" } new_table_name: "default.Table_6eb43dd09ca247e99abdb0335957a77f" modify_external_catalogs: true
I20250110 02:51:11.157658 1357 tablet.cc:1721] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:11.170562 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b (DEFAULT_TABLE table=default.Table_87db4b399a114ebd9397992a26db0aba [id=62df87bfc41d477fb6044eff0ed5c3ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.172128 1351 raft_consensus.cc:491] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.171898 745 tablet_service.cc:1467] Processing CreateTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 (DEFAULT_TABLE table=default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.172295 1351 raft_consensus.cc:513] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.172260 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:11.172907 776 raft_consensus.cc:3054] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.173885 1351 leader_election.cc:290] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
W20250110 02:51:11.174587 709 consensus_peers.cc:487] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80900124f93d43a0abf93175b16225a7. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:11.174997 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:11.175071 749 tablet_service.cc:1467] Processing CreateTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.175235 775 raft_consensus.cc:2463] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:11.177836 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0d22fd75fb474ebf07b624499a6f7b (DEFAULT_TABLE table=default.Table_accf846f8cb6482db39436051e3fdd6f [id=1d00dbb015e142329070adad93cf1bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.179795 1179 log.cc:1198] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b851ddf27fe44d19b9ca39d82e6352cd
I20250110 02:51:11.180235 1179 ts_tablet_manager.cc:1939] T b851ddf27fe44d19b9ca39d82e6352cd P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.180321 996 leader_election.cc:304] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.180385 1181 tablet_bootstrap.cc:654] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.180603 1031 tablet_service.cc:1467] Processing CreateTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 (DEFAULT_TABLE table=default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.180812 1038 tablet_service.cc:1467] Processing CreateTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa (DEFAULT_TABLE table=default.Table_a0cd879bafe848c6bb8e40d099e6b735 [id=488d922a6b604812af65ff6b4d9c6606]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.180999 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc (DEFAULT_TABLE table=default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.181203 1179 tablet_replica.cc:331] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.181455 1179 raft_consensus.cc:2238] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.181855 1179 pending_rounds.cc:62] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:11.181919 1179 pending_rounds.cc:69] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611717543333888 op_type: NO_OP noop_request { }
I20250110 02:51:11.182070 1179 raft_consensus.cc:2883] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.182178 1179 raft_consensus.cc:2267] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.182353 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:11.182901 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb5412b78104d8085878c58b191ab27 (DEFAULT_TABLE table=default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.182968 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e8b2297377bc4e5095826e30523eede7 (DEFAULT_TABLE table=default.Table_0e304e93af93489cae36fca1cfd0fd8d [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.183276 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 (DEFAULT_TABLE table=default.Table_8da521fca5664959b1c0801b30199a6f [id=4893b36e085445c4a454cf47ebe1955b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.183575 1061 raft_consensus.cc:2463] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:11.183853 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0b91f351e02d406f8675eb44c4c04ced (DEFAULT_TABLE table=default.Table_7489371158674f9abe9b0cafbd4f1157 [id=dfb29df1e3c54611abc07a3326df3215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.184170 1024 tablet_service.cc:1467] Processing CreateTablet for tablet da8f247c862e4cb68452e230857ee6b0 (DEFAULT_TABLE table=default.Table_bc03ee1fa76e458c8029c902c813dcb2 [id=e2a6764b09814b9cb1367abb1a4f5ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.184543 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 (DEFAULT_TABLE table=default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.184727 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 (DEFAULT_TABLE table=default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.185101 1351 raft_consensus.cc:2798] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:11.185212 1351 raft_consensus.cc:491] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:11.185304 1351 raft_consensus.cc:3054] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.187032 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0b91f351e02d406f8675eb44c4c04ced (DEFAULT_TABLE table=default.Table_7489371158674f9abe9b0cafbd4f1157 [id=dfb29df1e3c54611abc07a3326df3215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.187249 753 tablet_service.cc:1467] Processing CreateTablet for tablet e8b2297377bc4e5095826e30523eede7 (DEFAULT_TABLE table=default.Table_0e304e93af93489cae36fca1cfd0fd8d [id=c501975e25c34817b2c02a0e538c97a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.187597 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb5412b78104d8085878c58b191ab27 (DEFAULT_TABLE table=default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.188041 756 tablet_service.cc:1467] Processing CreateTablet for tablet da8f247c862e4cb68452e230857ee6b0 (DEFAULT_TABLE table=default.Table_bc03ee1fa76e458c8029c902c813dcb2 [id=e2a6764b09814b9cb1367abb1a4f5ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.188386 1351 raft_consensus.cc:513] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.188669 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 (DEFAULT_TABLE table=default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.188948 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b851ddf27fe44d19b9ca39d82e6352cd (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
I20250110 02:51:11.188987 740 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 (DEFAULT_TABLE table=default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.189147 746 tablet_service.cc:1467] Processing CreateTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc (DEFAULT_TABLE table=default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.189639 741 tablet_service.cc:1467] Processing CreateTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 (DEFAULT_TABLE table=default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.190302 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f6f4f221e6445db0bb569f372cfaa0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:11.190142 776 raft_consensus.cc:2463] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:11.190539 1060 raft_consensus.cc:3054] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.191164 1179 ts_tablet_manager.cc:1905] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.193547 1060 raft_consensus.cc:2463] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:11.194875 711 consensus_peers.cc:487] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c845c27a86d4e4f93c932313ebf648b. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:11.195511 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:11.195732 776 raft_consensus.cc:3054] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.200290 1351 leader_election.cc:290] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.202116 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3352f8378b9846a3854086293f2d4e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.202806 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e72ed2d31a8467f8c092b48d593c1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.203028 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec1dba54a247d5ba9f9e053aa3b645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.203265 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14f06ba7d0f44418693e36f6bc6d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.203472 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad0d561a8ef498ba13bc813d4cd82fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.206530 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e183122d13a4c38988bf2fe6f7a3694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.206730 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b91f351e02d406f8675eb44c4c04ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.206995 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ae1c19d2dc4778a1cebddbd505f907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.207427 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b2297377bc4e5095826e30523eede7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.210489 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb5412b78104d8085878c58b191ab27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.212670 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef72e05f8ed4d46b759c7a04497653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.213680 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0d22fd75fb474ebf07b624499a6f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.215457 1180 tablet_replica.cc:331] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:11.217463 1180 raft_consensus.cc:2238] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.217962 1180 raft_consensus.cc:2267] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.227775 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e534bf66f044b88d8cda0f3a1228f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.231410 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8f247c862e4cb68452e230857ee6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.236802 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af3d61366b2432db53b18f2b7bb49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.245467 776 raft_consensus.cc:2463] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:11.253803 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_73580b26ac6a47ab81519905e8adfb2f" } modify_external_catalogs: true
I20250110 02:51:11.254110 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_73580b26ac6a47ab81519905e8adfb2f" } modify_external_catalogs: true
I20250110 02:51:11.264153 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585320c0970c4097b195caaae33789b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.264505 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e534bf66f044b88d8cda0f3a1228f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.264710 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.264920 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60486a3dbc5a48eaa26735b84ae3fd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.265112 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a4745356e3423f9d4892d242da1c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.270874 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:11.272243 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:11.274441 882 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 (DEFAULT_TABLE table=default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.276515 853 leader_election.cc:304] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.277455 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e183122d13a4c38988bf2fe6f7a3694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.278126 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf711522866d46439d7ee69aa63d7f39. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.279497 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 82636.7us
I20250110 02:51:11.284127 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b2297377bc4e5095826e30523eede7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.285751 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb5412b78104d8085878c58b191ab27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.286424 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8f247c862e4cb68452e230857ee6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.288578 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef72e05f8ed4d46b759c7a04497653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.291837 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec1dba54a247d5ba9f9e053aa3b645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.292367 1184 tablet_bootstrap.cc:492] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.293449 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e744cdc1fcb94d46b48b684763ecf1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.295286 1184 ts_tablet_manager.cc:1397] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.429s user 0.077s sys 0.140s
I20250110 02:51:11.297986 1184 raft_consensus.cc:357] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.298573 1184 raft_consensus.cc:383] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.298676 1184 raft_consensus.cc:738] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.299561 1184 consensus_queue.cc:260] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.300293 1184 ts_tablet_manager.cc:1428] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:11.306475 1184 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:11.311949 1198 raft_consensus.cc:2798] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:11.320618 1314 raft_consensus.cc:491] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.320796 1314 raft_consensus.cc:513] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.322685 1314 leader_election.cc:290] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:11.322767 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:11.323961 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:11.324303 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250110 02:51:11.328595 1184 tablet_bootstrap.cc:654] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.333918 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3352f8378b9846a3854086293f2d4e15. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.338709 709 consensus_peers.cc:487] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c845c27a86d4e4f93c932313ebf648b. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:11.338718 894 tablet_service.cc:1467] Processing CreateTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 (DEFAULT_TABLE table=default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.344550 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:11.351377 1179 ts_tablet_manager.cc:1918] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.351536 1179 log.cc:1198] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/80900124f93d43a0abf93175b16225a7
I20250110 02:51:11.362548 1179 ts_tablet_manager.cc:1939] T 80900124f93d43a0abf93175b16225a7 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:11.362720 1241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 02:51:11.363095 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:11.363296 1241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 02:51:11.364053 1241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 02:51:11.383559 1184 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.383736 1184 ts_tablet_manager.cc:1397] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.011s
I20250110 02:51:11.384617 1184 raft_consensus.cc:357] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.384861 1184 raft_consensus.cc:383] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.384967 1184 raft_consensus.cc:738] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.385262 1184 consensus_queue.cc:260] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.386627 1184 ts_tablet_manager.cc:1428] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250110 02:51:11.404731 918 raft_consensus.cc:2463] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:11.405290 709 leader_election.cc:304] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.405588 1314 raft_consensus.cc:2798] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:11.405700 1314 raft_consensus.cc:491] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:11.405812 1314 raft_consensus.cc:3054] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.405822 1241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=34089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:11.408596 1314 raft_consensus.cc:513] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.409283 1314 leader_election.cc:290] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:11.410243 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:11.410491 918 raft_consensus.cc:3054] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.412086 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:11.413166 918 raft_consensus.cc:2463] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:11.413637 709 leader_election.cc:304] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.414402 1314 raft_consensus.cc:2798] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:11.418552 1314 raft_consensus.cc:695] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:11.418977 1314 consensus_queue.cc:237] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:11.419698 510 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0b74812cb81c44fcb8786cc5dae328e3 (table default.Table_0e304e93af93489cae36fca1cfd0fd8d [id=c501975e25c34817b2c02a0e538c97a7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0b74812cb81c44fcb8786cc5dae328e3 consensus state has no leader
I20250110 02:51:11.422873 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ae1c19d2dc4778a1cebddbd505f907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.422431 1241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250110 02:51:11.439102 1366 raft_consensus.cc:491] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.439324 1366 raft_consensus.cc:513] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.439635 1198 raft_consensus.cc:695] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:11.441663 1198 consensus_queue.cc:237] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.455701 1366 leader_election.cc:290] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:11.474470 885 consensus_meta.cc:287] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Time spent flushing consensus metadata: real 0.503s user 0.005s sys 0.000s
I20250110 02:51:11.218616 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e744cdc1fcb94d46b48b684763ecf1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.475322 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e183122d13a4c38988bf2fe6f7a3694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.219924 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 (DEFAULT_TABLE table=default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.475543 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b91f351e02d406f8675eb44c4c04ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.475767 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec1dba54a247d5ba9f9e053aa3b645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.476078 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef72e05f8ed4d46b759c7a04497653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.246193 996 leader_election.cc:304] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:11.478531 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 232329us
I20250110 02:51:11.322958 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:11.479312 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:11.480675 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 106) took 1055 ms. Trace:
I20250110 02:51:11.414147 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:11.455781 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f404a2e7494645af449aea3d3d3bf0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:11.480813 1179 rpcz_store.cc:276] 0110 02:51:10.425589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.425649 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:11.181113 (+755464us) ts_tablet_manager.cc:1145] Deleting tablet 80900124f93d43a0abf93175b16225a7
0110 02:51:11.181121 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.347971 (+166850us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.351269 (+ 3298us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.480593 (+129324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126720,"tablet-delete.queue_time_us":749923}
I20250110 02:51:11.481417 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 105) took 1057 ms. Trace:
I20250110 02:51:11.481508 1179 rpcz_store.cc:276] 0110 02:51:10.423562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.423638 (+ 76us) service_pool.cc:225] Handling call
0110 02:51:11.481238 (+1057600us) ts_tablet_manager.cc:1145] Deleting tablet 70af6ec40c7a4442b91680d8125c2d48
0110 02:51:11.481247 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.481358 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046721}
I20250110 02:51:11.481841 1179 tablet_replica.cc:331] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.482030 1179 raft_consensus.cc:2238] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.482757 1184 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:11.482698 853 leader_election.cc:304] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:11.484884 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 80900124f93d43a0abf93175b16225a7 (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
W20250110 02:51:11.485188 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 70af6ec40c7a4442b91680d8125c2d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70af6ec40c7a4442b91680d8125c2d48 already in progress: creating tablet
I20250110 02:51:11.485736 1061 raft_consensus.cc:2463] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:11.486518 994 leader_election.cc:343] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:11.487516 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 215588us
W20250110 02:51:11.487840 994 leader_election.cc:343] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:11.488404 1352 raft_consensus.cc:491] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.488489 1179 pending_rounds.cc:62] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:11.488561 1179 pending_rounds.cc:69] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611717456470016 op_type: NO_OP noop_request { }
I20250110 02:51:11.488567 1352 raft_consensus.cc:513] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.489584 1179 raft_consensus.cc:2883] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.489737 1179 raft_consensus.cc:2267] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.489807 1352 leader_election.cc:290] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.490237 1351 raft_consensus.cc:2798] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:11.490552 1179 ts_tablet_manager.cc:1905] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.491362 1351 raft_consensus.cc:695] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:11.492000 1351 consensus_queue.cc:237] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.492647 1184 tablet_bootstrap.cc:654] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.493726 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fb5412b78104d8085878c58b191ab27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.494611 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da8f247c862e4cb68452e230857ee6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.495182 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e72ed2d31a8467f8c092b48d593c1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.496827 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67adef4c3f4041e1ab480100ca762f62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:11.497035 776 raft_consensus.cc:2463] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:11.502398 1179 ts_tablet_manager.cc:1918] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:11.506624 1188 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.004s sys 0.001s
W20250110 02:51:11.506819 1188 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.532s user 0.004s sys 0.002s
W20250110 02:51:11.507896 1357 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.350s user 0.000s sys 0.000s
I20250110 02:51:11.508628 1179 log.cc:1198] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/4c845c27a86d4e4f93c932313ebf648b
I20250110 02:51:11.509086 1179 ts_tablet_manager.cc:1939] T 4c845c27a86d4e4f93c932313ebf648b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.509923 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 111) took 1054 ms. Trace:
I20250110 02:51:11.510041 1179 rpcz_store.cc:276] 0110 02:51:10.455810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.455853 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:11.481754 (+1025901us) ts_tablet_manager.cc:1145] Deleting tablet 4c845c27a86d4e4f93c932313ebf648b
0110 02:51:11.481763 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.498922 (+ 17159us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.502276 (+ 3354us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.509858 (+ 7582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024190}
I20250110 02:51:11.512526 1184 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.512692 1184 ts_tablet_manager.cc:1397] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250110 02:51:11.513517 1184 raft_consensus.cc:357] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.513777 1184 raft_consensus.cc:383] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.513873 1184 raft_consensus.cc:738] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.514115 1184 consensus_queue.cc:260] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.514984 1184 ts_tablet_manager.cc:1428] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:11.515256 1184 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:11.516634 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a4745356e3423f9d4892d242da1c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.517133 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 107) took 1088 ms. Trace:
I20250110 02:51:11.517277 1179 rpcz_store.cc:276] 0110 02:51:10.428253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.428323 (+ 70us) service_pool.cc:225] Handling call
0110 02:51:11.516949 (+1088626us) ts_tablet_manager.cc:1145] Deleting tablet ad9f920cde954f31acebc6d0b6965e58
0110 02:51:11.516959 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.517078 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057632}
W20250110 02:51:11.517794 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet ad9f920cde954f31acebc6d0b6965e58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad9f920cde954f31acebc6d0b6965e58 already in progress: creating tablet
I20250110 02:51:11.518263 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 110) took 1068 ms. Trace:
I20250110 02:51:11.518405 1179 rpcz_store.cc:276] 0110 02:51:10.450223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.450292 (+ 69us) service_pool.cc:225] Handling call
0110 02:51:11.518106 (+1067814us) ts_tablet_manager.cc:1145] Deleting tablet aa40d20288644b2a8d36ede967d16e79
0110 02:51:11.518117 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.518216 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049464}
W20250110 02:51:11.518903 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet aa40d20288644b2a8d36ede967d16e79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa40d20288644b2a8d36ede967d16e79 already in progress: creating tablet
I20250110 02:51:11.519755 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ce1eaf1a53cd4620a73549f2734f168b" } modify_external_catalogs: true
I20250110 02:51:11.520140 1179 tablet_replica.cc:331] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.520058 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ce1eaf1a53cd4620a73549f2734f168b" } modify_external_catalogs: true
I20250110 02:51:11.520370 1179 raft_consensus.cc:2238] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.520539 1179 raft_consensus.cc:2267] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.520998 1179 ts_tablet_manager.cc:1905] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.521235 996 leader_election.cc:304] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.523475 1180 ts_tablet_manager.cc:1905] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.523725 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585320c0970c4097b195caaae33789b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.525095 1352 raft_consensus.cc:2798] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:11.529587 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 585320c0970c4097b195caaae33789b5
I20250110 02:51:11.530335 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2ca408d476aa49068e9ddf9a597b2e13" } modify_external_catalogs: true
I20250110 02:51:11.530529 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2ca408d476aa49068e9ddf9a597b2e13" } modify_external_catalogs: true
I20250110 02:51:11.532197 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aafff2e0de548129d99e9f07dc3ab11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:11.533517 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8b2297377bc4e5095826e30523eede7 consensus state has no leader
I20250110 02:51:11.533946 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b59340acab754f4abb5f772e7574a520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.534461 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b59340acab754f4abb5f772e7574a520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:11.535640 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67adef4c3f4041e1ab480100ca762f62 consensus state has no leader
I20250110 02:51:11.536062 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_85038c2c4ac94a6c81067f58f4e11e02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.536604 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85038c2c4ac94a6c81067f58f4e11e02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.538887 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.544363 1352 raft_consensus.cc:491] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:11.544515 1352 raft_consensus.cc:3054] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:11.546758 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:39025, user_credentials={real_user=slave}} blocked reactor thread for 62380.9us
I20250110 02:51:11.554183 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.559392 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.559453 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af3d61366b2432db53b18f2b7bb49d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.560560 1351 raft_consensus.cc:491] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.531567 1366 raft_consensus.cc:2743] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:11.540511 1188 tablet_bootstrap.cc:492] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:11.561635 1314 consensus_queue.cc:1035] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:11.564463 1179 ts_tablet_manager.cc:1918] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.564641 1179 log.cc:1198] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/10504e77a5c64ebf85be937d64ca9c80
I20250110 02:51:11.565061 1179 ts_tablet_manager.cc:1939] T 10504e77a5c64ebf85be937d64ca9c80 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.565804 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 109) took 1123 ms. Trace:
I20250110 02:51:11.565984 1179 rpcz_store.cc:276] 0110 02:51:10.442095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.450565 (+ 8470us) service_pool.cc:225] Handling call
0110 02:51:11.520079 (+1069514us) ts_tablet_manager.cc:1145] Deleting tablet 10504e77a5c64ebf85be937d64ca9c80
0110 02:51:11.520087 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.535848 (+ 15761us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.564359 (+ 28511us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.565771 (+ 1412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046934}
I20250110 02:51:11.566656 1179 tablet_replica.cc:331] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.566900 1179 raft_consensus.cc:2238] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.567049 1179 raft_consensus.cc:2267] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.569348 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 67adef4c3f4041e1ab480100ca762f62 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]). Expected version 3 got 0
I20250110 02:51:11.571780 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.573624 1188 ts_tablet_manager.cc:1397] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.636s user 0.006s sys 0.003s
I20250110 02:51:11.576742 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.577121 1179 ts_tablet_manager.cc:1905] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.578547 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14f06ba7d0f44418693e36f6bc6d888. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.578759 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:11.581054 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8823b1f4fcd7469c8d4eb6264e068b2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.581610 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.584647 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad0d561a8ef498ba13bc813d4cd82fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.584859 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0587dd5e537540b4911487707ec14464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.585347 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0587dd5e537540b4911487707ec14464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.583245 1351 raft_consensus.cc:513] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.587857 1351 leader_election.cc:290] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.587941 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.588171 748 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:11.590601 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60e534bf66f044b88d8cda0f3a1228f4 consensus state has no leader
I20250110 02:51:11.591125 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:11.591849 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:11.592046 776 raft_consensus.cc:2463] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:11.592576 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 67adef4c3f4041e1ab480100ca762f62 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:11.592799 996 leader_election.cc:304] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.593587 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ae1c19d2dc4778a1cebddbd505f907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.549093 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67adef4c3f4041e1ab480100ca762f62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:11.544225 890 consensus_meta.cc:287] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Time spent flushing consensus metadata: real 0.539s user 0.002s sys 0.002s
I20250110 02:51:11.597282 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.597476 1184 tablet_bootstrap.cc:654] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.600706 1178 ts_tablet_manager.cc:1918] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.601514 1178 log.cc:1198] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:11.601150 518 catalog_manager.cc:5526] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:11.601869 1178 ts_tablet_manager.cc:1939] T 2a27f0327d1844b8a561af8b0bf4385b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:11.602660 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:11.607373 1351 raft_consensus.cc:2798] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:11.610395 1184 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.610545 1184 ts_tablet_manager.cc:1397] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.004s
I20250110 02:51:11.611238 1314 raft_consensus.cc:491] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.611469 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.612082 1351 raft_consensus.cc:491] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:11.612440 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e534bf66f044b88d8cda0f3a1228f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.612051 1314 raft_consensus.cc:513] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.614176 1352 raft_consensus.cc:513] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.611398 1184 raft_consensus.cc:357] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.614449 1184 raft_consensus.cc:383] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.614539 1184 raft_consensus.cc:738] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.614862 1352 leader_election.cc:290] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.614820 1184 consensus_queue.cc:260] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:11.616442 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67adef4c3f4041e1ab480100ca762f62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:11.616670 776 raft_consensus.cc:3054] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:11.616758 709 leader_election.cc:343] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:11.617055 1184 ts_tablet_manager.cc:1428] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:11.617487 1184 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:11.618911 1314 leader_election.cc:290] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.620990 1184 tablet_bootstrap.cc:654] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.621296 776 raft_consensus.cc:2463] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:11.622128 994 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:11.622610 994 leader_election.cc:343] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:11.625677 1179 ts_tablet_manager.cc:1918] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.628448 1179 log.cc:1198] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/66f6f4f221e6445db0bb569f372cfaa0
I20250110 02:51:11.628804 1179 ts_tablet_manager.cc:1939] T 66f6f4f221e6445db0bb569f372cfaa0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.622969 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67adef4c3f4041e1ab480100ca762f62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:11.629612 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 117) took 1046 ms. Trace:
I20250110 02:51:11.629707 1179 rpcz_store.cc:276] 0110 02:51:10.583292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.583452 (+ 160us) service_pool.cc:225] Handling call
0110 02:51:11.566571 (+983119us) ts_tablet_manager.cc:1145] Deleting tablet 66f6f4f221e6445db0bb569f372cfaa0
0110 02:51:11.566580 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.591671 (+ 25091us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.594323 (+ 2652us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.629550 (+ 35227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965193}
I20250110 02:51:11.630038 1351 raft_consensus.cc:3054] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.630137 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 66f6f4f221e6445db0bb569f372cfaa0 (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
I20250110 02:51:11.631640 890 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.634168 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0d22fd75fb474ebf07b624499a6f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.634894 996 leader_election.cc:304] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:11.635636 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.635761 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 118) took 1052 ms. Trace:
I20250110 02:51:11.635845 1179 rpcz_store.cc:276] 0110 02:51:10.583407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.583624 (+ 217us) service_pool.cc:225] Handling call
0110 02:51:11.635582 (+1051958us) ts_tablet_manager.cc:1145] Deleting tablet c65704c9d486444ebe8e4ffd4580f395
0110 02:51:11.635593 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.635709 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1034159}
W20250110 02:51:11.636307 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c65704c9d486444ebe8e4ffd4580f395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65704c9d486444ebe8e4ffd4580f395 already in progress: creating tablet
I20250110 02:51:11.637385 1179 tablet_replica.cc:331] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.637445 1181 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:11.637642 1179 raft_consensus.cc:2238] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.637861 1179 raft_consensus.cc:2267] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.637866 1352 raft_consensus.cc:2798] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:11.638029 1352 raft_consensus.cc:695] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:11.639365 1179 ts_tablet_manager.cc:1905] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:11.640019 711 leader_election.cc:343] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:11.640204 711 leader_election.cc:304] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:11.640693 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9cf200cbed2644eba78cf579cad06583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.641227 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cf200cbed2644eba78cf579cad06583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.643101 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4422eca5604d149e73c2a669d07af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.643410 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_87db4b399a114ebd9397992a26db0aba" } new_table_name: "default.Table_e298b60e8778429bb52cf546d58fa2d9" modify_external_catalogs: true
I20250110 02:51:11.643708 1181 ts_tablet_manager.cc:1397] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.552s user 0.013s sys 0.001s
I20250110 02:51:11.644479 1181 raft_consensus.cc:357] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.644680 1181 raft_consensus.cc:383] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.644749 1181 raft_consensus.cc:738] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.649413 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:11.650056 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 8940c201aa104b0b8bc8d866859fe8ab from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:11.650271 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.651975 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 308f307f4200409aaa47df55f20472d0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:11.652707 1352 consensus_queue.cc:237] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.653396 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b2297377bc4e5095826e30523eede7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.654146 1181 consensus_queue.cc:260] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.654668 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.655969 1369 raft_consensus.cc:2743] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:11.657604 1369 consensus_queue.cc:1035] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:11.664182 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 308f307f4200409aaa47df55f20472d0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:11.665817 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_660c31e76a8c4a8ea394d251dd4718f6" } new_table_name: "default.Table_7881112a09664881843ffe986bb6aac2" modify_external_catalogs: true
I20250110 02:51:11.666417 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.657604 1179 ts_tablet_manager.cc:1918] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:11.668052 1179 log.cc:1198] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/866513b2738844bba19bb91a7f4bfd81
I20250110 02:51:11.668994 1179 ts_tablet_manager.cc:1939] T 866513b2738844bba19bb91a7f4bfd81 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.669703 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 115) took 1086 ms. Trace:
I20250110 02:51:11.669799 1179 rpcz_store.cc:276] 0110 02:51:10.583119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.583865 (+ 746us) service_pool.cc:225] Handling call
0110 02:51:11.637289 (+1053424us) ts_tablet_manager.cc:1145] Deleting tablet 866513b2738844bba19bb91a7f4bfd81
0110 02:51:11.637301 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.642205 (+ 4904us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.657449 (+ 15244us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.669667 (+ 12218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035757}
I20250110 02:51:11.670253 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]) successfully deleted
I20250110 02:51:11.672260 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 10ab917bde534e3fbdf5dbae4af9304e from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:11.676174 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 113) took 1093 ms. Trace:
I20250110 02:51:11.676294 1179 rpcz_store.cc:276] 0110 02:51:10.582884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.586299 (+ 3415us) service_pool.cc:225] Handling call
0110 02:51:11.676006 (+1089707us) ts_tablet_manager.cc:1145] Deleting tablet 89f6797b1076468a93fb0b510c553bed
0110 02:51:11.676018 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.676132 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074413}
W20250110 02:51:11.677922 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 89f6797b1076468a93fb0b510c553bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f6797b1076468a93fb0b510c553bed already in progress: creating tablet
I20250110 02:51:11.678360 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 114) took 1095 ms. Trace:
I20250110 02:51:11.678531 1179 rpcz_store.cc:276] 0110 02:51:10.583035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.587886 (+ 4851us) service_pool.cc:225] Handling call
0110 02:51:11.678197 (+1090311us) ts_tablet_manager.cc:1145] Deleting tablet 1dfc062b5fcb42c3bce6dfaed54b3869
0110 02:51:11.678209 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.678323 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076377}
I20250110 02:51:11.682299 1179 tablet_replica.cc:331] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.682579 1179 raft_consensus.cc:2238] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.682763 1179 raft_consensus.cc:2267] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.682619 1351 raft_consensus.cc:513] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.683370 1179 ts_tablet_manager.cc:1905] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.684576 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:11.686165 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c9d791b1c504171ba66dec55c8fb0e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:11.689906 994 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:11.690128 996 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: Tablet not RUNNING: STOPPED
I20250110 02:51:11.690806 996 leader_election.cc:304] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:11.691159 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e85cbad8a6e24bf59145c42e3b94ebf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.691632 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.694428 1351 leader_election.cc:290] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.695900 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 67adef4c3f4041e1ab480100ca762f62
I20250110 02:51:11.696314 519 catalog_manager.cc:6596] ReplaceTablet: tablet 67adef4c3f4041e1ab480100ca762f62 deleted and replaced by tablet 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:11.696662 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 67adef4c3f4041e1ab480100ca762f62 with delete_type TABLET_DATA_DELETED (replaced by tablet 09f01daf4c67462ba75e1d0cf56cef12) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:11.696748 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ef72e05f8ed4d46b759c7a04497653b consensus state has no leader
I20250110 02:51:11.699339 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 308f307f4200409aaa47df55f20472d0
I20250110 02:51:11.699779 522 catalog_manager.cc:6596] ReplaceTablet: tablet 308f307f4200409aaa47df55f20472d0 deleted and replaced by tablet aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:11.699898 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 308f307f4200409aaa47df55f20472d0 with delete_type TABLET_DATA_DELETED (replaced by tablet aa6057de9da04f2c82b8f25c5b4e2fe4) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:11.701452 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3ae68e4f0254ef38aafd5a1f8f88a54 has no consensus state
I20250110 02:51:11.701912 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:11.702436 524 catalog_manager.cc:6596] ReplaceTablet: tablet 8940c201aa104b0b8bc8d866859fe8ab deleted and replaced by tablet b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:11.703049 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.707181 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:11.709981 1387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 02:51:11.710212 1390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:11.714512 1241 server_base.cc:1034] running on GCE node
I20250110 02:51:11.716198 1351 raft_consensus.cc:2743] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 02:51:11.716949 888 tablet_service.cc:1467] Processing CreateTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 (DEFAULT_TABLE table=default.Table_5b2da56ee56b45bebccadc94ae4578a2 [id=8488c24625ca46ba8f0d2405ab565feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.717893 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d (DEFAULT_TABLE table=default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.718847 881 tablet_service.cc:1467] Processing CreateTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (DEFAULT_TABLE table=default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.720356 743 tablet_service.cc:1467] Processing CreateTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 (DEFAULT_TABLE table=default.Table_5b2da56ee56b45bebccadc94ae4578a2 [id=8488c24625ca46ba8f0d2405ab565feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.720361 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d10ace37b96844328ccdb821dad3c5ea (DEFAULT_TABLE table=default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.720572 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0127de45264f4660b843734a530eb310 (DEFAULT_TABLE table=default.Table_28491d38435d448b9ecbb83b5efb94d4 [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.720695 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66603a0ee8c4082807c5a6aabe7e339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.721047 748 tablet_service.cc:1467] Processing CreateTablet for tablet d10ace37b96844328ccdb821dad3c5ea (DEFAULT_TABLE table=default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.721393 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d10ace37b96844328ccdb821dad3c5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.721549 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d10ace37b96844328ccdb821dad3c5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.724164 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.724340 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d (DEFAULT_TABLE table=default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.724654 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558a2c03f6b42a6befdc48e3af23a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.726729 746 tablet_service.cc:1467] Processing CreateTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (DEFAULT_TABLE table=default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.727059 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ae68e4f0254ef38aafd5a1f8f88a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.727078 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 308f307f4200409aaa47df55f20472d0 with delete_type TABLET_DATA_DELETED (replaced by tablet aa6057de9da04f2c82b8f25c5b4e2fe4) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.730598 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0127de45264f4660b843734a530eb310 (DEFAULT_TABLE table=default.Table_28491d38435d448b9ecbb83b5efb94d4 [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.730873 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0127de45264f4660b843734a530eb310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.732808 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:39025: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111),
I20250110 02:51:11.732985 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250110 02:51:11.718317 1179 ts_tablet_manager.cc:1918] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.733484 1179 log.cc:1198] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/4c9d791b1c504171ba66dec55c8fb0e6
I20250110 02:51:11.733882 1179 ts_tablet_manager.cc:1939] T 4c9d791b1c504171ba66dec55c8fb0e6 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.734946 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 116) took 1151 ms. Trace:
I20250110 02:51:11.735049 1179 rpcz_store.cc:276] 0110 02:51:10.583210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.601210 (+ 18000us) service_pool.cc:225] Handling call
0110 02:51:11.682227 (+1081017us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d791b1c504171ba66dec55c8fb0e6
0110 02:51:11.682238 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.706130 (+ 23892us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.718214 (+ 12084us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.734870 (+ 16656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078833}
I20250110 02:51:11.735517 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 4c9d791b1c504171ba66dec55c8fb0e6 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
W20250110 02:51:11.735965 1380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:11.737185 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:11.726220 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d (DEFAULT_TABLE table=default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.739317 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 0127de45264f4660b843734a530eb310 (DEFAULT_TABLE table=default.Table_28491d38435d448b9ecbb83b5efb94d4 [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.740312 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558a2c03f6b42a6befdc48e3af23a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.741099 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0127de45264f4660b843734a530eb310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.742722 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 109) took 1021 ms. Trace:
I20250110 02:51:11.742761 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f404a2e7494645af449aea3d3d3bf0 with delete_type TABLET_DATA_DELETED (replaced by tablet c65704c9d486444ebe8e4ffd4580f395) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.742802 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.743355 1179 tablet_replica.cc:331] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.743567 1179 raft_consensus.cc:2238] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.742872 1178 rpcz_store.cc:276] 0110 02:51:10.599099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599158 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:11.001315 (+402157us) ts_tablet_manager.cc:1145] Deleting tablet 2a27f0327d1844b8a561af8b0bf4385b
0110 02:51:11.001336 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.589714 (+588378us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.600609 (+ 10895us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.609329 (+ 8720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":401907}
I20250110 02:51:11.743911 1178 tablet_replica.cc:331] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:11.743978 1179 pending_rounds.cc:62] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:11.744050 1179 pending_rounds.cc:69] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611716251353088 op_type: NO_OP noop_request { }
I20250110 02:51:11.744108 1178 raft_consensus.cc:2238] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.744215 1179 raft_consensus.cc:2883] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.744333 1179 raft_consensus.cc:2267] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.744351 1178 pending_rounds.cc:62] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Trying to abort 1 pending ops.
I20250110 02:51:11.744413 1178 pending_rounds.cc:69] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611723030093824 op_type: NO_OP noop_request { }
I20250110 02:51:11.744554 1178 raft_consensus.cc:2883] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.744647 1178 raft_consensus.cc:2267] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.744858 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 67adef4c3f4041e1ab480100ca762f62 with delete_type TABLET_DATA_DELETED (replaced by tablet 09f01daf4c67462ba75e1d0cf56cef12) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.745162 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 308f307f4200409aaa47df55f20472d0 with delete_type TABLET_DATA_DELETED (replaced by tablet aa6057de9da04f2c82b8f25c5b4e2fe4) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.745179 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2a27f0327d1844b8a561af8b0bf4385b (table default.Table_cbaca8b1d598406db88d9fcef46dd17b [id=b74f16dd5b58413c8dff82451dd83cd6]) successfully deleted
I20250110 02:51:11.745456 885 tablet_service.cc:1467] Processing CreateTablet for tablet d10ace37b96844328ccdb821dad3c5ea (DEFAULT_TABLE table=default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.745715 1178 ts_tablet_manager.cc:1905] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.745883 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66603a0ee8c4082807c5a6aabe7e339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.746212 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0127de45264f4660b843734a530eb310. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.746901 1179 ts_tablet_manager.cc:1905] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.750514 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d10ace37b96844328ccdb821dad3c5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.752636 1178 ts_tablet_manager.cc:1918] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.752769 1178 log.cc:1198] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/66f6f4f221e6445db0bb569f372cfaa0
I20250110 02:51:11.753147 1178 ts_tablet_manager.cc:1939] T 66f6f4f221e6445db0bb569f372cfaa0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:11.753298 1179 ts_tablet_manager.cc:1918] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.753427 1179 log.cc:1198] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d58e92b9789e49f6bbfbefe3f8f3a0ef
I20250110 02:51:11.753811 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ae68e4f0254ef38aafd5a1f8f88a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.753880 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 110) took 1154 ms. Trace:
I20250110 02:51:11.753976 1178 rpcz_store.cc:276] 0110 02:51:10.599273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599339 (+ 66us) service_pool.cc:225] Handling call
0110 02:51:11.743821 (+1144482us) ts_tablet_manager.cc:1145] Deleting tablet 66f6f4f221e6445db0bb569f372cfaa0
0110 02:51:11.743831 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.749508 (+ 5677us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.752550 (+ 3042us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.753818 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144238}
I20250110 02:51:11.745920 1188 raft_consensus.cc:357] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.754475 1188 raft_consensus.cc:383] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.754565 1188 raft_consensus.cc:738] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.754556 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 66f6f4f221e6445db0bb569f372cfaa0 (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
I20250110 02:51:11.754779 1188 consensus_queue.cc:260] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:11.755033 918 raft_consensus.cc:3054] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.755185 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558a2c03f6b42a6befdc48e3af23a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.757812 1188 ts_tablet_manager.cc:1428] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.183s user 0.003s sys 0.000s
I20250110 02:51:11.758083 918 raft_consensus.cc:2463] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:11.760294 1188 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:11.761384 1179 ts_tablet_manager.cc:1939] T d58e92b9789e49f6bbfbefe3f8f3a0ef P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.761561 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 112) took 1162 ms. Trace:
I20250110 02:51:11.761675 1178 rpcz_store.cc:276] 0110 02:51:10.599558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599614 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:11.761382 (+1161768us) ts_tablet_manager.cc:1145] Deleting tablet 1dfc062b5fcb42c3bce6dfaed54b3869
0110 02:51:11.761393 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.761505 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161534}
I20250110 02:51:11.762075 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 108) took 1331 ms. Trace:
I20250110 02:51:11.762152 1179 rpcz_store.cc:276] 0110 02:51:10.430666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.454529 (+ 23863us) service_pool.cc:225] Handling call
0110 02:51:11.743264 (+1288735us) ts_tablet_manager.cc:1145] Deleting tablet d58e92b9789e49f6bbfbefe3f8f3a0ef
0110 02:51:11.743274 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.750312 (+ 7038us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.753214 (+ 2902us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.762028 (+ 8814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126538}
I20250110 02:51:11.762557 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d58e92b9789e49f6bbfbefe3f8f3a0ef (table default.Table_f0abf378d8b94be9b42028b3ef26bbe5 [id=b94c35cd0cd3480cb751f1d841850f4a]) successfully deleted
I20250110 02:51:11.762801 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:11.763831 1188 tablet_bootstrap.cc:654] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.763770 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1621908f975348db834b77c8e899eb67" } new_table_name: "default.Table_be1a53ef30a94d35ad3de81d4e9effee" modify_external_catalogs: true
W20250110 02:51:11.764681 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4e2c01af512f460daa98b19444f72172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2c01af512f460daa98b19444f72172 already in progress: creating tablet
I20250110 02:51:11.765156 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f8b1849990d84955a6b754fdf58919d4" } new_table_name: "default.Table_306f9f127f75420882d16d81923bf9f5" modify_external_catalogs: true
W20250110 02:51:11.765467 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0c2412420dc54258a6921ead194c1587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c2412420dc54258a6921ead194c1587 already in progress: creating tablet
W20250110 02:51:11.766196 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 423c820b81b845719d59a376c18e175a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423c820b81b845719d59a376c18e175a already in progress: creating tablet
W20250110 02:51:11.766933 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2abd85a070f74e9bb5dc4026935edcb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2abd85a070f74e9bb5dc4026935edcb5 already in progress: creating tablet
I20250110 02:51:11.767374 1179 tablet_replica.cc:331] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.767565 1179 raft_consensus.cc:2238] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.766304 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_85038c2c4ac94a6c81067f58f4e11e02" } modify_external_catalogs: true
I20250110 02:51:11.768311 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_85038c2c4ac94a6c81067f58f4e11e02" } modify_external_catalogs: true
I20250110 02:51:11.771045 1178 tablet_replica.cc:331] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:11.771204 1178 raft_consensus.cc:2238] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:11.763439 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.772282 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 019c0d60da5548bb86c93ccd89bf94ff
W20250110 02:51:11.774123 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f229f546a8d4a09b93f8b46c893d14a consensus state has no leader
I20250110 02:51:11.775358 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_570c693b36a44e53b70e27098df8de91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.775895 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_570c693b36a44e53b70e27098df8de91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.776918 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.777258 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.780251 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b7a2bc15b1114d39944fadd733f2eb4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.780699 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7a2bc15b1114d39944fadd733f2eb4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.781548 1184 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.781757 1184 ts_tablet_manager.cc:1397] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.164s user 0.034s sys 0.016s
I20250110 02:51:11.783721 524 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0b6658fd21c143f39d42abdb7d6b4c4c
I20250110 02:51:11.784143 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7cebc36aba3d43e0877447c797c945f7" } modify_external_catalogs: true
I20250110 02:51:11.784322 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7cebc36aba3d43e0877447c797c945f7" } modify_external_catalogs: true
I20250110 02:51:11.767753 1179 pending_rounds.cc:62] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:11.787878 1179 pending_rounds.cc:69] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611719745671168 op_type: NO_OP noop_request { }
I20250110 02:51:11.788036 1179 raft_consensus.cc:2883] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.788157 1179 raft_consensus.cc:2267] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.789557 1314 raft_consensus.cc:491] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:11.789765 1314 raft_consensus.cc:513] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.789935 1184 raft_consensus.cc:357] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.790102 1184 raft_consensus.cc:383] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.790171 1184 raft_consensus.cc:738] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.790448 1314 leader_election.cc:290] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:11.790413 1184 consensus_queue.cc:260] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.790930 1184 ts_tablet_manager.cc:1428] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250110 02:51:11.791247 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:11.791937 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f14f06ba7d0f44418693e36f6bc6d888
I20250110 02:51:11.792914 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.795374 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_cea75045b3d94a008b85e2f3bbefdebc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.795825 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea75045b3d94a008b85e2f3bbefdebc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.796276 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5b2da56ee56b45bebccadc94ae4578a2" } new_table_name: "default.Table_7ab0055319bd4962a0267ce290806e0f" modify_external_catalogs: true
I20250110 02:51:11.798782 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_163c6a20e5ad4f8398dc3a947c92bf9f" } modify_external_catalogs: true
I20250110 02:51:11.798975 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_163c6a20e5ad4f8398dc3a947c92bf9f" } modify_external_catalogs: true
I20250110 02:51:11.799134 1179 ts_tablet_manager.cc:1905] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.800288 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.801574 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.802706 887 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.802858 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aafff2e0de548129d99e9f07dc3ab11" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:11.803085 918 raft_consensus.cc:2388] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98336b684b41407da8a1949e30cd82e0 in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:11.803519 709 leader_election.cc:304] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:11.806046 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (DEFAULT_TABLE table=default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.806844 1029 tablet_service.cc:1467] Processing CreateTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 (DEFAULT_TABLE table=default.Table_5b2da56ee56b45bebccadc94ae4578a2 [id=8488c24625ca46ba8f0d2405ab565feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.807237 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66603a0ee8c4082807c5a6aabe7e339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.808084 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 67adef4c3f4041e1ab480100ca762f62 with delete_type TABLET_DATA_DELETED (replaced by tablet 09f01daf4c67462ba75e1d0cf56cef12) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.809408 1184 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:11.811717 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f66603a0ee8c4082807c5a6aabe7e339 consensus state has no leader
I20250110 02:51:11.814317 1184 tablet_bootstrap.cc:654] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:11.816704 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ae68e4f0254ef38aafd5a1f8f88a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.817530 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.818409 1314 raft_consensus.cc:3054] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:11.820665 749 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc89802926543738cb04c75821ab693 (DEFAULT_TABLE table=default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 [id=056af6d890614cae954fa6f9e3f7f6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.820722 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc89802926543738cb04c75821ab693 (DEFAULT_TABLE table=default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 [id=056af6d890614cae954fa6f9e3f7f6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.820914 890 tablet_service.cc:1467] Processing CreateTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 (DEFAULT_TABLE table=default.Table_b59340acab754f4abb5f772e7574a520 [id=996c474b298141e48b7a828a367d0e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.821027 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc89802926543738cb04c75821ab693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.821094 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc89802926543738cb04c75821ab693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.822178 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d4d6e02b1d4334a9993d320217f1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.822626 886 tablet_service.cc:1467] Processing CreateTablet for tablet b3a9eaf548274b73a5652ba314dab70c (DEFAULT_TABLE table=default.Table_9cf200cbed2644eba78cf579cad06583 [id=58c2b267860e44998d49c1905758f326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.822942 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a9eaf548274b73a5652ba314dab70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.826710 748 tablet_service.cc:1467] Processing CreateTablet for tablet b3a9eaf548274b73a5652ba314dab70c (DEFAULT_TABLE table=default.Table_9cf200cbed2644eba78cf579cad06583 [id=58c2b267860e44998d49c1905758f326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.826800 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc89802926543738cb04c75821ab693 (DEFAULT_TABLE table=default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 [id=056af6d890614cae954fa6f9e3f7f6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.827046 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a9eaf548274b73a5652ba314dab70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.827124 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc89802926543738cb04c75821ab693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.829497 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:11.830902 1241 hybrid_clock.cc:648] HybridClock initialized: now 1736477471830796 us; error 90 us; skew 500 ppm
I20250110 02:51:11.831364 1241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:11.835012 748 tablet_service.cc:1467] Processing CreateTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 (DEFAULT_TABLE table=default.Table_b59340acab754f4abb5f772e7574a520 [id=996c474b298141e48b7a828a367d0e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.835820 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b3a9eaf548274b73a5652ba314dab70c (DEFAULT_TABLE table=default.Table_9cf200cbed2644eba78cf579cad06583 [id=58c2b267860e44998d49c1905758f326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.836203 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a9eaf548274b73a5652ba314dab70c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.841598 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 48cb20d47ace4fd49f9dc4e526cab702
I20250110 02:51:11.847735 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.848265 749 tablet_service.cc:1467] Processing CreateTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.848740 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce95413df40369ee599176f91b892 (DEFAULT_TABLE table=default.Table_0587dd5e537540b4911487707ec14464 [id=0743ba60cabd49c7a2c5230109d764eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.848740 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7902775b8db247a89d9b20d0ee880e36 (DEFAULT_TABLE table=default.Table_cf93380c055c4e4c8c9fe4e359e5b874 [id=a68859689efc4362ab519c103dfc42a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.848794 746 tablet_service.cc:1467] Processing CreateTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 (DEFAULT_TABLE table=default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.848623 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d4d6e02b1d4334a9993d320217f1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.849251 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f01daf4c67462ba75e1d0cf56cef12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.849478 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7902775b8db247a89d9b20d0ee880e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.849877 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3ce95413df40369ee599176f91b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.851451 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1dc0c1e6f84c4bff8bc4c249ac502084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.851961 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc0c1e6f84c4bff8bc4c249ac502084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.852006 1038 tablet_service.cc:1467] Processing CreateTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 (DEFAULT_TABLE table=default.Table_b59340acab754f4abb5f772e7574a520 [id=996c474b298141e48b7a828a367d0e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.853029 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d4d6e02b1d4334a9993d320217f1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.853224 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_48581fba686f44bea7a67b12a11e149f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:11.853279 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ca9f3a9eeaa845498cf76982cc9b59fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.853694 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48581fba686f44bea7a67b12a11e149f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:11.853760 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca9f3a9eeaa845498cf76982cc9b59fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.856030 1379 consensus_queue.cc:1035] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:11.856911 1314 raft_consensus.cc:2743] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:11.857048 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e676f1148b5f4988a2be33981eec57c3" } modify_external_catalogs: true
I20250110 02:51:11.857228 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e676f1148b5f4988a2be33981eec57c3" } modify_external_catalogs: true
I20250110 02:51:11.859004 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.862620 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.863020 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.851608 1179 ts_tablet_manager.cc:1918] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.866670 1179 log.cc:1198] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/05d5f9bda55942ff856252bbb50b3c5b
I20250110 02:51:11.867264 1179 ts_tablet_manager.cc:1939] T 05d5f9bda55942ff856252bbb50b3c5b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.871256 1179 tablet_replica.cc:331] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.871440 1179 raft_consensus.cc:2238] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:11.871680 1179 pending_rounds.cc:62] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:11.871747 1179 pending_rounds.cc:69] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611722941009920 op_type: NO_OP noop_request { }
I20250110 02:51:11.871901 1179 raft_consensus.cc:2883] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:11.872004 1179 raft_consensus.cc:2267] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.872599 1179 ts_tablet_manager.cc:1905] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.872733 894 tablet_service.cc:1467] Processing CreateTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.873114 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.875836 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_57b14b339ca74ec597640f14ceed500f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.876355 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57b14b339ca74ec597640f14ceed500f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.879029 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7902775b8db247a89d9b20d0ee880e36 (DEFAULT_TABLE table=default.Table_cf93380c055c4e4c8c9fe4e359e5b874 [id=a68859689efc4362ab519c103dfc42a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.879462 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7902775b8db247a89d9b20d0ee880e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.882517 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.882601 740 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7885a80414f2193400672aa16a911 (DEFAULT_TABLE table=default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.882892 880 tablet_service.cc:1467] Processing CreateTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 (DEFAULT_TABLE table=default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.883183 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f01daf4c67462ba75e1d0cf56cef12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.882936 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a7885a80414f2193400672aa16a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.883471 756 tablet_service.cc:1467] Processing CreateTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.883195 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4778347f054b86af697dbe3b287263 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.884027 743 tablet_service.cc:1467] Processing CreateTablet for tablet 820975aa259f4274ba6c19398425e94b (DEFAULT_TABLE table=default.Table_d8da12d0416f480794b1d69b6c62f386 [id=7d20fea78e494fe3bc3631bb749d58eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.884174 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2163c343a694f5ca04ac9f7610d25e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.884469 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820975aa259f4274ba6c19398425e94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.886708 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce95413df40369ee599176f91b892 (DEFAULT_TABLE table=default.Table_0587dd5e537540b4911487707ec14464 [id=0743ba60cabd49c7a2c5230109d764eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.887012 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3ce95413df40369ee599176f91b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.889036 744 tablet_service.cc:1467] Processing CreateTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 (DEFAULT_TABLE table=default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.889446 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69d0c201c524b7e95c9d3851b8dd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.889516 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]) successfully deleted
I20250110 02:51:11.892561 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 64c77ec8c7724d5fb408a8bfa53811eb
I20250110 02:51:11.892624 879 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7885a80414f2193400672aa16a911 (DEFAULT_TABLE table=default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.892935 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a7885a80414f2193400672aa16a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.902177 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 7902775b8db247a89d9b20d0ee880e36 (DEFAULT_TABLE table=default.Table_cf93380c055c4e4c8c9fe4e359e5b874 [id=a68859689efc4362ab519c103dfc42a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.902554 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7902775b8db247a89d9b20d0ee880e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.903545 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_564f2d190f904222b9f705210a88ac5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.904062 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_564f2d190f904222b9f705210a88ac5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.904878 1179 ts_tablet_manager.cc:1918] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:11.905002 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:11.907775 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 (DEFAULT_TABLE table=default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.908144 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09f01daf4c67462ba75e1d0cf56cef12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.913348 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3ce95413df40369ee599176f91b892 (DEFAULT_TABLE table=default.Table_0587dd5e537540b4911487707ec14464 [id=0743ba60cabd49c7a2c5230109d764eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.905005 1179 log.cc:1198] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a9f404a2e7494645af449aea3d3d3bf0
I20250110 02:51:11.913973 1179 ts_tablet_manager.cc:1939] T a9f404a2e7494645af449aea3d3d3bf0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.916536 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 134) took 1046 ms. Trace:
I20250110 02:51:11.916625 1179 rpcz_store.cc:276] 0110 02:51:10.870077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.871240 (+ 1163us) service_pool.cc:225] Handling call
0110 02:51:11.871165 (+999925us) ts_tablet_manager.cc:1145] Deleting tablet a9f404a2e7494645af449aea3d3d3bf0
0110 02:51:11.871173 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.902306 (+ 31133us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.904788 (+ 2482us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.916485 (+ 11697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999743}
I20250110 02:51:11.916954 1179 tablet_replica.cc:331] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.917114 1179 raft_consensus.cc:2238] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.917258 1179 raft_consensus.cc:2267] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.917750 1179 ts_tablet_manager.cc:1905] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.919656 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a9f404a2e7494645af449aea3d3d3bf0 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:11.920331 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4778347f054b86af697dbe3b287263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.921732 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3ce95413df40369ee599176f91b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.913724 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7885a80414f2193400672aa16a911 (DEFAULT_TABLE table=default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.923136 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2500bc74e2d44275bab206e0ba400d75" } new_table_name: "default.Table_6c7eac9722834633b607ad9b97e5947d" modify_external_catalogs: true
I20250110 02:51:11.923066 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1a56c4b5a83345798b5d0f524ba0560d" } new_table_name: "default.Table_11510771ec434825b06bd79a30ce92b8" modify_external_catalogs: true
I20250110 02:51:11.923395 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a7885a80414f2193400672aa16a911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.923259 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_db04ddcfa7864a5a82f05df887a858aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.923763 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db04ddcfa7864a5a82f05df887a858aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.926013 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:11.930011 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00618f611d4c4a84a988125c5d175efc consensus state has no leader
I20250110 02:51:11.931926 749 tablet_service.cc:1467] Processing CreateTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (DEFAULT_TABLE table=default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.932308 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa6057de9da04f2c82b8f25c5b4e2fe4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.935073 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60486a3dbc5a48eaa26735b84ae3fd6f consensus state has no leader
I20250110 02:51:11.802579 1178 raft_consensus.cc:2267] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.937325 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:11.915421 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.915970 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4778347f054b86af697dbe3b287263 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.937943 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4778347f054b86af697dbe3b287263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.938612 897 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4778347f054b86af697dbe3b287263 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.940016 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4778347f054b86af697dbe3b287263. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.917618 1041 tablet_service.cc:1467] Processing CreateTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (DEFAULT_TABLE table=default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.919389 1180 ts_tablet_manager.cc:1918] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:11.917639 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 820975aa259f4274ba6c19398425e94b (DEFAULT_TABLE table=default.Table_d8da12d0416f480794b1d69b6c62f386 [id=7d20fea78e494fe3bc3631bb749d58eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.920522 1037 tablet_service.cc:1467] Processing CreateTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 (DEFAULT_TABLE table=default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.948704 1179 ts_tablet_manager.cc:1918] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:11.948825 881 tablet_service.cc:1467] Processing CreateTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (DEFAULT_TABLE table=default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.949183 1178 ts_tablet_manager.cc:1905] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:11.950729 1181 ts_tablet_manager.cc:1428] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.307s user 0.011s sys 0.003s
I20250110 02:51:11.950754 1179 log.cc:1198] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2a27f0327d1844b8a561af8b0bf4385b
I20250110 02:51:11.951560 1179 ts_tablet_manager.cc:1939] T 2a27f0327d1844b8a561af8b0bf4385b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:11.952874 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 135) took 1078 ms. Trace:
I20250110 02:51:11.953822 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:11.958686 1241 webserver.cc:458] Webserver started at http://127.0.104.60:34089/ using document root <none> and password file <none>
I20250110 02:51:11.958652 898 tablet_service.cc:1467] Processing CreateTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 (DEFAULT_TABLE table=default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.959897 1241 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:11.960120 1241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:11.960805 890 tablet_service.cc:1467] Processing CreateTablet for tablet 820975aa259f4274ba6c19398425e94b (DEFAULT_TABLE table=default.Table_d8da12d0416f480794b1d69b6c62f386 [id=7d20fea78e494fe3bc3631bb749d58eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.954409 1179 rpcz_store.cc:276] 0110 02:51:10.874105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.878706 (+ 4601us) service_pool.cc:225] Handling call
0110 02:51:11.916880 (+1038174us) ts_tablet_manager.cc:1145] Deleting tablet 2a27f0327d1844b8a561af8b0bf4385b
0110 02:51:11.916888 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.937875 (+ 20987us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.948589 (+ 10714us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.952832 (+ 4243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035861}
I20250110 02:51:11.965624 1188 tablet_bootstrap.cc:492] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:11.967468 1180 log.cc:1198] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/866513b2738844bba19bb91a7f4bfd81
I20250110 02:51:11.967996 1180 ts_tablet_manager.cc:1939] T 866513b2738844bba19bb91a7f4bfd81 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:11.968263 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2a27f0327d1844b8a561af8b0bf4385b (table default.Table_cbaca8b1d598406db88d9fcef46dd17b [id=b74f16dd5b58413c8dff82451dd83cd6]) successfully deleted
I20250110 02:51:11.968264 886 tablet_service.cc:1467] Processing CreateTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.962734 1184 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:11.968886 1184 ts_tablet_manager.cc:1397] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.160s user 0.007s sys 0.006s
I20250110 02:51:11.971303 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa6057de9da04f2c82b8f25c5b4e2fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.971393 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 142) took 1035 ms. Trace:
I20250110 02:51:11.971494 1179 rpcz_store.cc:276] 0110 02:51:10.936266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.941449 (+ 5183us) service_pool.cc:225] Handling call
0110 02:51:11.971246 (+1029797us) ts_tablet_manager.cc:1145] Deleting tablet 0b74812cb81c44fcb8786cc5dae328e3
0110 02:51:11.971255 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.971362 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029602}
I20250110 02:51:11.971963 1179 tablet_replica.cc:331] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:11.971911 1184 raft_consensus.cc:357] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.972110 1184 raft_consensus.cc:383] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:11.972164 1179 raft_consensus.cc:2238] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:11.972205 1184 raft_consensus.cc:738] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:11.972316 1179 raft_consensus.cc:2267] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:11.972411 1184 consensus_queue.cc:260] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:11.972860 1179 ts_tablet_manager.cc:1905] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:11.974118 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0b74812cb81c44fcb8786cc5dae328e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b74812cb81c44fcb8786cc5dae328e3 already in progress: creating tablet
I20250110 02:51:11.974977 523 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:11.975569 1184 ts_tablet_manager.cc:1428] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:11.975914 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b59340acab754f4abb5f772e7574a520" } new_table_name: "default.Table_a4b11eff25ef415cbdd62ed46e34ddcf" modify_external_catalogs: true
I20250110 02:51:11.977236 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cf93380c055c4e4c8c9fe4e359e5b874" } new_table_name: "default.Table_2bc2ba61f91d4e01ad04875d5e28c79e" modify_external_catalogs: true
I20250110 02:51:11.978502 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69d0c201c524b7e95c9d3851b8dd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.979521 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa6057de9da04f2c82b8f25c5b4e2fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.980362 1184 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:11.983960 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_88abc71d734244088bac8733189eae55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:11.983384 749 tablet_service.cc:1467] Processing CreateTablet for tablet d9018955f3884824beb2e7336987fc1c (DEFAULT_TABLE table=default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.984094 1026 tablet_service.cc:1467] Processing CreateTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 (DEFAULT_TABLE table=default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:11.984419 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88abc71d734244088bac8733189eae55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.985002 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9018955f3884824beb2e7336987fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.985620 740 tablet_service.cc:1467] Processing CreateTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 (DEFAULT_TABLE table=default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.985796 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:11.985994 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc9a4a842914a6f878d01194cd9a5f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:11.986847 994 consensus_peers.cc:487] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:11.988080 1188 ts_tablet_manager.cc:1397] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.206s user 0.005s sys 0.001s
I20250110 02:51:11.988423 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7881112a09664881843ffe986bb6aac2" } modify_external_catalogs: true
I20250110 02:51:11.988587 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7881112a09664881843ffe986bb6aac2" } modify_external_catalogs: true
I20250110 02:51:11.991115 748 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6de957fd14af983fc1a3cd8629226 (DEFAULT_TABLE table=default.Table_1dc0c1e6f84c4bff8bc4c249ac502084 [id=bca53af91a9746b990b7329b337d993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.991509 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c6de957fd14af983fc1a3cd8629226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.991622 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_64d72c69126e4192abb401ad9e536c0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.992096 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d72c69126e4192abb401ad9e536c0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:11.993228 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69d0c201c524b7e95c9d3851b8dd8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:11.994009 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ff78b6f4c98a42b69c23212d1fa6f09b" } modify_external_catalogs: true
I20250110 02:51:11.994160 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ff78b6f4c98a42b69c23212d1fa6f09b" } modify_external_catalogs: true
I20250110 02:51:11.994549 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_57b14b339ca74ec597640f14ceed500f" } modify_external_catalogs: true
I20250110 02:51:11.994719 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_57b14b339ca74ec597640f14ceed500f" } modify_external_catalogs: true
I20250110 02:51:11.997629 885 tablet_service.cc:1467] Processing CreateTablet for tablet d9018955f3884824beb2e7336987fc1c (DEFAULT_TABLE table=default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:11.997751 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f8ccf39d30184c9fad3eb52f047115cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:11.998160 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0d4d6e02b1d4334a9993d320217f1f8 consensus state has no leader
W20250110 02:51:11.998224 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8ccf39d30184c9fad3eb52f047115cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:11.998801 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7902775b8db247a89d9b20d0ee880e36 consensus state has no leader
I20250110 02:51:12.004776 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f9dbb61a51f14a3f96cc3c77981e8ce5
I20250110 02:51:12.007014 882 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6de957fd14af983fc1a3cd8629226 (DEFAULT_TABLE table=default.Table_1dc0c1e6f84c4bff8bc4c249ac502084 [id=bca53af91a9746b990b7329b337d993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.008671 1184 tablet_bootstrap.cc:654] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.010205 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.008926 1188 raft_consensus.cc:357] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.010869 748 tablet_service.cc:1467] Processing CreateTablet for tablet 397f3261b76a4c82a3c265e68662841f (DEFAULT_TABLE table=default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 [id=88d6d3dc5db2480aa3ec55acb2ec5cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.011039 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:12.011397 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 397f3261b76a4c82a3c265e68662841f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.011389 743 tablet_service.cc:1467] Processing CreateTablet for tablet f6df074974974d81a83420e7a3a78070 (DEFAULT_TABLE table=default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.012359 888 tablet_service.cc:1467] Processing CreateTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 (DEFAULT_TABLE table=default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.012858 895 tablet_service.cc:1467] Processing CreateTablet for tablet 397f3261b76a4c82a3c265e68662841f (DEFAULT_TABLE table=default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 [id=88d6d3dc5db2480aa3ec55acb2ec5cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.013564 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820975aa259f4274ba6c19398425e94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.017454 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.020350 1188 raft_consensus.cc:383] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.020448 1188 raft_consensus.cc:738] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.020897 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 820975aa259f4274ba6c19398425e94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.021277 1188 consensus_queue.cc:260] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.022053 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 397f3261b76a4c82a3c265e68662841f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.023537 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.023967 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df074974974d81a83420e7a3a78070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.025354 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c6de957fd14af983fc1a3cd8629226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.030490 1181 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:12.030514 1034 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.030836 1038 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6de957fd14af983fc1a3cd8629226 (DEFAULT_TABLE table=default.Table_1dc0c1e6f84c4bff8bc4c249ac502084 [id=bca53af91a9746b990b7329b337d993b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.031179 1032 tablet_service.cc:1467] Processing CreateTablet for tablet d9018955f3884824beb2e7336987fc1c (DEFAULT_TABLE table=default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.032070 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 111) took 1435 ms. Trace:
I20250110 02:51:12.032214 1180 rpcz_store.cc:276] 0110 02:51:10.576712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.576760 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:11.087663 (+510903us) ts_tablet_manager.cc:1145] Deleting tablet 866513b2738844bba19bb91a7f4bfd81
0110 02:51:11.087672 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:11.629601 (+541929us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:11.919306 (+289705us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.012604 (+ 93298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1665,"tablet-delete.queue_time_us":510662}
I20250110 02:51:12.033651 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 110) took 1457 ms. Trace:
I20250110 02:51:12.033766 1180 rpcz_store.cc:276] 0110 02:51:10.576582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.577195 (+ 613us) service_pool.cc:225] Handling call
0110 02:51:12.033414 (+1456219us) ts_tablet_manager.cc:1145] Deleting tablet 1dfc062b5fcb42c3bce6dfaed54b3869
0110 02:51:12.033428 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.033590 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456011}
I20250110 02:51:12.034168 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 112) took 1453 ms. Trace:
I20250110 02:51:12.034549 1386 raft_consensus.cc:491] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.034688 1386 raft_consensus.cc:513] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.035434 1386 leader_election.cc:290] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:12.036108 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc9a4a842914a6f878d01194cd9a5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.035722 892 tablet_service.cc:1467] Processing CreateTablet for tablet f6df074974974d81a83420e7a3a78070 (DEFAULT_TABLE table=default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.039151 1181 tablet_bootstrap.cc:654] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.039664 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9018955f3884824beb2e7336987fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.039626 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_018e09f0d16349738140c8eef71e9050" } new_table_name: "default.Table_31a8cb16aea94ae7ae6370b386b9dd82" modify_external_catalogs: true
I20250110 02:51:12.041954 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 397f3261b76a4c82a3c265e68662841f (DEFAULT_TABLE table=default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 [id=88d6d3dc5db2480aa3ec55acb2ec5cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.042307 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 397f3261b76a4c82a3c265e68662841f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.043798 1036 tablet_service.cc:1467] Processing CreateTablet for tablet f6df074974974d81a83420e7a3a78070 (DEFAULT_TABLE table=default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.044131 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df074974974d81a83420e7a3a78070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.046440 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_dce75d4fa88e4bf59fe18cf4725d953f" } modify_external_catalogs: true
I20250110 02:51:12.046649 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_dce75d4fa88e4bf59fe18cf4725d953f" } modify_external_catalogs: true
I20250110 02:51:12.050683 1188 ts_tablet_manager.cc:1428] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.051s user 0.001s sys 0.003s
I20250110 02:51:12.050464 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a65eb3bd7f2c46ed98d0890044cdf125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.053182 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a65eb3bd7f2c46ed98d0890044cdf125 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.053213 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.055617 517 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]). Expected version 1 got 0
I20250110 02:51:12.034258 1180 rpcz_store.cc:276] 0110 02:51:10.581015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.581065 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:12.034020 (+1452955us) ts_tablet_manager.cc:1145] Deleting tablet c65704c9d486444ebe8e4ffd4580f395
0110 02:51:12.034029 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.034108 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1452790}
I20250110 02:51:12.058768 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]) successfully deleted
W20250110 02:51:12.059378 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c65704c9d486444ebe8e4ffd4580f395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65704c9d486444ebe8e4ffd4580f395 already in progress: creating tablet
I20250110 02:51:12.062029 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9018955f3884824beb2e7336987fc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.064265 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2163c343a694f5ca04ac9f7610d25e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.067253 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.068665 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 109) took 1489 ms. Trace:
I20250110 02:51:12.069588 1180 rpcz_store.cc:276] 0110 02:51:10.575897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.600951 (+ 25054us) service_pool.cc:225] Handling call
0110 02:51:12.057801 (+1456850us) ts_tablet_manager.cc:1145] Deleting tablet 89f6797b1076468a93fb0b510c553bed
0110 02:51:12.057813 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.065219 (+ 7406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456663}
I20250110 02:51:12.070405 1180 tablet_replica.cc:331] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:12.070345 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 89f6797b1076468a93fb0b510c553bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f6797b1076468a93fb0b510c553bed already in progress: creating tablet
I20250110 02:51:12.070729 1180 raft_consensus.cc:2238] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.071365 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c6de957fd14af983fc1a3cd8629226. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.071779 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ab917bde534e3fbdf5dbae4af9304e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.071965 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_39db324aabe548a28775a241e962cfe7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.072690 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39db324aabe548a28775a241e962cfe7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.073122 525 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]). Expected version 1 got 0
I20250110 02:51:12.080529 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcc9a4a842914a6f878d01194cd9a5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.080523 776 raft_consensus.cc:1270] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Refusing update from remote peer 8899dd1aeb384e4ea4523f5fd225c888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:12.081312 1180 raft_consensus.cc:2267] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:12.083392 994 leader_election.cc:343] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.083559 1379 consensus_queue.cc:1035] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250110 02:51:12.084208 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2163c343a694f5ca04ac9f7610d25e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.086637 740 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.086870 1180 ts_tablet_manager.cc:1905] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.083886 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ab917bde534e3fbdf5dbae4af9304e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:12.087834 776 raft_consensus.cc:2463] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:12.087975 1188 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:12.088271 996 leader_election.cc:304] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:12.088698 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df074974974d81a83420e7a3a78070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.089455 1379 raft_consensus.cc:2798] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:12.089558 1379 raft_consensus.cc:491] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:12.096704 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 02d3f60c819f4824bc99d755e7442976 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.096784 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b064f366436b4b72bc8428d0bbb0c3df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.097330 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b064f366436b4b72bc8428d0bbb0c3df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.099056 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d62161a3f0a2401198ea6728f350a9f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.099478 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d62161a3f0a2401198ea6728f350a9f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.103185 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_570c693b36a44e53b70e27098df8de91" } modify_external_catalogs: true
I20250110 02:51:12.103384 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_570c693b36a44e53b70e27098df8de91" } modify_external_catalogs: true
I20250110 02:51:12.105650 1184 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.105854 1184 ts_tablet_manager.cc:1397] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.003s
I20250110 02:51:12.106016 1379 raft_consensus.cc:3054] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.113648 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 10ab917bde534e3fbdf5dbae4af9304e from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.120393 1184 raft_consensus.cc:357] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.120730 1184 raft_consensus.cc:383] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.121274 1184 raft_consensus.cc:738] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.123096 526 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]). Expected version 1 got 0
I20250110 02:51:12.123941 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d133af1dd7834a27b1d3ee0e4e788fe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.124392 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.124660 1241 fs_manager.cc:714] Time spent opening directory manager: real 0.161s user 0.005s sys 0.006s
I20250110 02:51:12.126225 1184 consensus_queue.cc:260] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.128329 1184 ts_tablet_manager.cc:1428] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250110 02:51:12.129216 1184 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.127811 1179 ts_tablet_manager.cc:1918] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.130053 1179 log.cc:1198] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1aafff2e0de548129d99e9f07dc3ab11
I20250110 02:51:12.131494 1179 ts_tablet_manager.cc:1939] T 1aafff2e0de548129d99e9f07dc3ab11 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.132673 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1aafff2e0de548129d99e9f07dc3ab11 (table default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]) successfully deleted
I20250110 02:51:12.134202 1184 tablet_bootstrap.cc:654] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:12.134816 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 585320c0970c4097b195caaae33789b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585320c0970c4097b195caaae33789b5 already in progress: creating tablet
I20250110 02:51:12.135293 1179 tablet_replica.cc:331] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:12.136286 1179 raft_consensus.cc:2238] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.136682 1179 raft_consensus.cc:2267] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.138875 520 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d95430bfdde849f79a30ae38ab4dca20
I20250110 02:51:12.140662 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6123629172f54edb8b4da1696d019100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.141134 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6123629172f54edb8b4da1696d019100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.142735 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_cccf34a241bd4aaf9296a023cdd9a853"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.143237 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cccf34a241bd4aaf9296a023cdd9a853 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.146823 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.150624 1379 raft_consensus.cc:513] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.151762 1379 leader_election.cc:290] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:12.152679 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ab917bde534e3fbdf5dbae4af9304e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:12.153254 996 leader_election.cc:343] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.156147 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ab917bde534e3fbdf5dbae4af9304e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:12.157927 1188 tablet_bootstrap.cc:654] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:12.158458 994 leader_election.cc:343] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.158609 994 leader_election.cc:304] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.161152 1393 raft_consensus.cc:2743] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.164106 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0587dd5e537540b4911487707ec14464" } new_table_name: "default.Table_a95ba565dbf940f79ed6f8b2eff1e74b" modify_external_catalogs: true
I20250110 02:51:12.165673 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8eb6afa433004d3aaac88418b07a91f3" } new_table_name: "default.Table_1c00825881c54f55b095b5854ea07eaa" modify_external_catalogs: true
I20250110 02:51:12.167227 1417 tablet.cc:1721] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:12.168268 1178 ts_tablet_manager.cc:1918] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:12.168399 1178 log.cc:1198] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/866513b2738844bba19bb91a7f4bfd81
I20250110 02:51:12.168910 1178 ts_tablet_manager.cc:1939] T 866513b2738844bba19bb91a7f4bfd81 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:12.169617 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 113) took 1569 ms. Trace:
I20250110 02:51:12.170346 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:12.170441 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d9018955f3884824beb2e7336987fc1c
I20250110 02:51:12.169745 1178 rpcz_store.cc:276] 0110 02:51:10.599724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599794 (+ 70us) service_pool.cc:225] Handling call
0110 02:51:11.767757 (+1167963us) ts_tablet_manager.cc:1145] Deleting tablet 866513b2738844bba19bb91a7f4bfd81
0110 02:51:11.767765 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.020700 (+252935us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.168129 (+147429us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.169586 (+ 1457us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167766}
I20250110 02:51:12.171238 1184 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.171238 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 114) took 1571 ms. Trace:
I20250110 02:51:12.171341 1178 rpcz_store.cc:276] 0110 02:51:10.599887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599922 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:12.171093 (+1571171us) ts_tablet_manager.cc:1145] Deleting tablet c65704c9d486444ebe8e4ffd4580f395
0110 02:51:12.171104 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.171207 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1570971}
I20250110 02:51:12.171646 1184 ts_tablet_manager.cc:1397] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20250110 02:51:12.175127 1179 ts_tablet_manager.cc:1905] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.175788 1180 ts_tablet_manager.cc:1918] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.175933 1180 log.cc:1198] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/886ab6d12cf545b9b700959e18396c50
I20250110 02:51:12.176295 1180 ts_tablet_manager.cc:1939] T 886ab6d12cf545b9b700959e18396c50 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:12.177297 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 111) took 1577 ms. Trace:
I20250110 02:51:12.177424 1178 rpcz_store.cc:276] 0110 02:51:10.599380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.599634 (+ 254us) service_pool.cc:225] Handling call
0110 02:51:12.177128 (+1577494us) ts_tablet_manager.cc:1145] Deleting tablet 89f6797b1076468a93fb0b510c553bed
0110 02:51:12.177139 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.177259 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576914}
I20250110 02:51:12.178501 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.178720 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 866513b2738844bba19bb91a7f4bfd81 (table default.Table_9c251b39e0f04bf7916d2ce12652063c [id=308c5c775cc34795af11c57c3593cc28]) successfully deleted
I20250110 02:51:12.179260 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.179576 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 120) took 1409 ms. Trace:
I20250110 02:51:12.179996 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.180450 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.179968 1178 rpcz_store.cc:276] 0110 02:51:10.770204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.775169 (+ 4965us) service_pool.cc:225] Handling call
0110 02:51:12.179476 (+1404307us) ts_tablet_manager.cc:1145] Deleting tablet c683bc3feaff44b1830cccfe839f677e
0110 02:51:12.179484 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.179544 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394221}
I20250110 02:51:12.180721 1184 raft_consensus.cc:357] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.180891 1181 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:12.180958 1184 raft_consensus.cc:383] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.181042 1181 ts_tablet_manager.cc:1397] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.230s user 0.019s sys 0.031s
I20250110 02:51:12.181068 1184 raft_consensus.cc:738] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.181211 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 121) took 1395 ms. Trace:
W20250110 02:51:12.181259 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6757b7ecc514489a0fa7e1ff18206bd consensus state has no leader
I20250110 02:51:12.181317 1178 rpcz_store.cc:276] 0110 02:51:10.785361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.789023 (+ 3662us) service_pool.cc:225] Handling call
0110 02:51:12.180889 (+1391866us) ts_tablet_manager.cc:1145] Deleting tablet 4e2c01af512f460daa98b19444f72172
0110 02:51:12.180897 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.181181 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323062}
I20250110 02:51:12.183678 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 124) took 1359 ms. Trace:
I20250110 02:51:12.183794 1178 rpcz_store.cc:276] 0110 02:51:10.822539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.823847 (+ 1308us) service_pool.cc:225] Handling call
0110 02:51:12.182084 (+1358237us) ts_tablet_manager.cc:1145] Deleting tablet 0c2412420dc54258a6921ead194c1587
0110 02:51:12.182096 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.182185 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1319002}
I20250110 02:51:12.184559 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 128) took 1316 ms. Trace:
I20250110 02:51:12.184676 1178 rpcz_store.cc:276] 0110 02:51:10.867868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.868042 (+ 174us) service_pool.cc:225] Handling call
0110 02:51:12.184429 (+1316387us) ts_tablet_manager.cc:1145] Deleting tablet 423c820b81b845719d59a376c18e175a
0110 02:51:12.184437 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.184532 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309759}
I20250110 02:51:12.185635 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.185853 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]). Expected version 1 got 0
I20250110 02:51:12.185920 1181 raft_consensus.cc:357] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:12.186826 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c65704c9d486444ebe8e4ffd4580f395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65704c9d486444ebe8e4ffd4580f395 already in progress: creating tablet
I20250110 02:51:12.187024 1181 raft_consensus.cc:383] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:12.187353 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 89f6797b1076468a93fb0b510c553bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f6797b1076468a93fb0b510c553bed already in progress: creating tablet
I20250110 02:51:12.187582 1181 raft_consensus.cc:738] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:12.187610 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c683bc3feaff44b1830cccfe839f677e because the tablet was not found. No further retry: Not found: Tablet not found: c683bc3feaff44b1830cccfe839f677e
W20250110 02:51:12.187850 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4e2c01af512f460daa98b19444f72172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2c01af512f460daa98b19444f72172 already in progress: creating tablet
W20250110 02:51:12.188205 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 423c820b81b845719d59a376c18e175a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423c820b81b845719d59a376c18e175a already in progress: creating tablet
I20250110 02:51:12.188876 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 118) took 1395 ms. Trace:
I20250110 02:51:12.190847 1184 consensus_queue.cc:260] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.191102 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.191534 1184 ts_tablet_manager.cc:1428] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
W20250110 02:51:12.192456 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c3ce95413df40369ee599176f91b892 consensus state has no leader
I20250110 02:51:12.194809 1418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:12.196708 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 129) took 1328 ms. Trace:
I20250110 02:51:12.196761 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]). Expected version 1 got 0
I20250110 02:51:12.196815 1178 rpcz_store.cc:276] 0110 02:51:10.868001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.868114 (+ 113us) service_pool.cc:225] Handling call
0110 02:51:12.196555 (+1328441us) ts_tablet_manager.cc:1145] Deleting tablet 2abd85a070f74e9bb5dc4026935edcb5
0110 02:51:12.196566 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.196676 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321711}
I20250110 02:51:12.198227 1178 tablet_replica.cc:331] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:12.198278 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd6d4c4631994a5792200ba4d4c906f0 consensus state has no leader
W20250110 02:51:12.198635 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2abd85a070f74e9bb5dc4026935edcb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2abd85a070f74e9bb5dc4026935edcb5 already in progress: creating tablet
I20250110 02:51:12.199012 1184 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.201323 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a0cd879bafe848c6bb8e40d099e6b735" } new_table_name: "default.Table_5bfa749055ab45479cf96da68e537ece" modify_external_catalogs: true
I20250110 02:51:12.202317 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a65eb3bd7f2c46ed98d0890044cdf125" } new_table_name: "default.Table_309dd7a424574e50a1d844d112efa307" modify_external_catalogs: true
W20250110 02:51:12.202945 996 consensus_peers.cc:487] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aafff2e0de548129d99e9f07dc3ab11. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:12.204010 1178 raft_consensus.cc:2238] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.188165 1181 consensus_queue.cc:260] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.204192 1178 pending_rounds.cc:62] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Trying to abort 1 pending ops.
I20250110 02:51:12.204269 1178 pending_rounds.cc:69] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611719745671168 op_type: NO_OP noop_request { }
I20250110 02:51:12.204382 1178 raft_consensus.cc:2883] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:12.204493 1178 raft_consensus.cc:2267] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.204618 749 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.206846 1181 ts_tablet_manager.cc:1428] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250110 02:51:12.207041 1178 ts_tablet_manager.cc:1905] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.196235 1180 rpcz_store.cc:276] 0110 02:51:10.793262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.793749 (+ 487us) service_pool.cc:225] Handling call
0110 02:51:12.070106 (+1276357us) ts_tablet_manager.cc:1145] Deleting tablet 886ab6d12cf545b9b700959e18396c50
0110 02:51:12.070116 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.165915 (+ 95799us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.175658 (+ 9743us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.188838 (+ 13180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276140}
I20250110 02:51:12.190966 1420 tablet.cc:1721] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:12.207940 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a77dd8b2a8d1473db5913fcea0cdce3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.208989 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 886ab6d12cf545b9b700959e18396c50 (table default.Table_1c00825881c54f55b095b5854ea07eaa [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]) successfully deleted
W20250110 02:51:12.208758 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.211201 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 119) took 1417 ms. Trace:
I20250110 02:51:12.211304 1180 rpcz_store.cc:276] 0110 02:51:10.793364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.793997 (+ 633us) service_pool.cc:225] Handling call
0110 02:51:12.207441 (+1413444us) ts_tablet_manager.cc:1145] Deleting tablet 4e2c01af512f460daa98b19444f72172
0110 02:51:12.207450 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.211162 (+ 3712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1413276}
I20250110 02:51:12.211606 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 123) took 1401 ms. Trace:
I20250110 02:51:12.211702 1180 rpcz_store.cc:276] 0110 02:51:10.809865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.811707 (+ 1842us) service_pool.cc:225] Handling call
0110 02:51:12.211489 (+1399782us) ts_tablet_manager.cc:1145] Deleting tablet 0c2412420dc54258a6921ead194c1587
0110 02:51:12.211496 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.211570 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1399463}
I20250110 02:51:12.211964 1180 tablet_replica.cc:331] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:12.212080 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.212157 1180 raft_consensus.cc:2238] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.212472 1180 raft_consensus.cc:2267] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:12.212965 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4e2c01af512f460daa98b19444f72172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2c01af512f460daa98b19444f72172 already in progress: creating tablet
I20250110 02:51:12.213006 1181 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:12.213358 1180 ts_tablet_manager.cc:1905] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.214498 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f8ccf39d30184c9fad3eb52f047115cb" } modify_external_catalogs: true
I20250110 02:51:12.214699 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f8ccf39d30184c9fad3eb52f047115cb" } modify_external_catalogs: true
I20250110 02:51:12.216068 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.216630 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.216944 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e298b60e8778429bb52cf546d58fa2d9" } modify_external_catalogs: true
I20250110 02:51:12.217097 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e298b60e8778429bb52cf546d58fa2d9" } modify_external_catalogs: true
I20250110 02:51:12.215111 1181 tablet_bootstrap.cc:654] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.221419 1184 tablet_bootstrap.cc:654] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.223197 1179 ts_tablet_manager.cc:1918] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.223325 1179 log.cc:1198] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/10ab917bde534e3fbdf5dbae4af9304e
I20250110 02:51:12.223726 1179 ts_tablet_manager.cc:1939] T 10ab917bde534e3fbdf5dbae4af9304e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.223969 1178 ts_tablet_manager.cc:1918] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:12.224079 1178 log.cc:1198] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/05d5f9bda55942ff856252bbb50b3c5b
I20250110 02:51:12.224454 1178 ts_tablet_manager.cc:1939] T 05d5f9bda55942ff856252bbb50b3c5b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:12.224756 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 130) took 1355 ms. Trace:
I20250110 02:51:12.224871 1178 rpcz_store.cc:276] 0110 02:51:10.868901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.868947 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:12.198154 (+1329207us) ts_tablet_manager.cc:1145] Deleting tablet 05d5f9bda55942ff856252bbb50b3c5b
0110 02:51:12.198165 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.221158 (+ 22993us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.223865 (+ 2707us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.224725 (+ 860us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323184}
I20250110 02:51:12.225301 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]) successfully deleted
I20250110 02:51:12.226168 1188 tablet_bootstrap.cc:492] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:12.226929 1188 ts_tablet_manager.cc:1397] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.155s user 0.008s sys 0.002s
I20250110 02:51:12.227926 1188 raft_consensus.cc:357] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.228191 1188 raft_consensus.cc:383] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.228312 1188 raft_consensus.cc:738] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.228607 1188 consensus_queue.cc:260] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.230024 1188 ts_tablet_manager.cc:1428] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:12.230413 1188 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:12.231925 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dad0d561a8ef498ba13bc813d4cd82fa consensus state has no leader
I20250110 02:51:12.231947 1188 tablet_bootstrap.cc:654] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:12.232350 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d5ddbc6b02742609bc2ca84c863a9d0 has no consensus state
I20250110 02:51:12.235802 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c9d791b1c504171ba66dec55c8fb0e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 8940c201aa104b0b8bc8d866859fe8ab) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.238971 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 10ab917bde534e3fbdf5dbae4af9304e (table default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]) successfully deleted
I20250110 02:51:12.239205 1179 tablet_replica.cc:331] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:12.239228 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b already in progress: creating tablet
W20250110 02:51:12.239445 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9e72ed2d31a8467f8c092b48d593c1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e72ed2d31a8467f8c092b48d593c1cc already in progress: creating tablet
I20250110 02:51:12.239446 1179 raft_consensus.cc:2238] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.239625 1179 raft_consensus.cc:2267] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:12.239697 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5af3d61366b2432db53b18f2b7bb49d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5af3d61366b2432db53b18f2b7bb49d9 already in progress: creating tablet
I20250110 02:51:12.240218 1179 ts_tablet_manager.cc:1905] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.240695 525 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 12c970753fb841218b9603e86a20cda2
I20250110 02:51:12.241142 1178 tablet_replica.cc:331] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:12.246364 1180 ts_tablet_manager.cc:1918] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.247129 1181 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:12.248548 1314 raft_consensus.cc:491] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.248674 1314 raft_consensus.cc:513] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.249274 1314 leader_election.cc:290] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.250029 749 tablet_service.cc:1467] Processing CreateTablet for tablet 871180863899419595cd632e7a4cf0dc (DEFAULT_TABLE table=default.Table_88abc71d734244088bac8733189eae55 [id=6f37f6d6318e45f1a016b8a34634050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.250586 743 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd370138ac4039be55642f1b4c5607 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.251210 1184 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.251392 756 tablet_service.cc:1467] Processing CreateTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (DEFAULT_TABLE table=default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.251569 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871180863899419595cd632e7a4cf0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.251706 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0588e5e5714cc5bd1c73665e5645b1 (DEFAULT_TABLE table=default.Table_ca9f3a9eeaa845498cf76982cc9b59fd [id=bda31fe0278a4b9a95eedb85d71b2eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.251803 744 tablet_service.cc:1467] Processing CreateTablet for tablet e3fc9826ef554a5699e365d10d02e97a (DEFAULT_TABLE table=default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.252034 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0588e5e5714cc5bd1c73665e5645b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.252089 1184 ts_tablet_manager.cc:1397] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.000s
I20250110 02:51:12.252251 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.252489 740 tablet_service.cc:1467] Processing CreateTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 (DEFAULT_TABLE table=default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.253332 1184 raft_consensus.cc:357] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.253551 1184 raft_consensus.cc:383] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.253638 1184 raft_consensus.cc:738] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.253930 1184 consensus_queue.cc:260] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.254434 1184 ts_tablet_manager.cc:1428] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:12.254753 1184 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.255141 522 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666] alter complete (version 1)
I20250110 02:51:12.256275 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fc9826ef554a5699e365d10d02e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.257318 1184 tablet_bootstrap.cc:654] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.249789 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2412420dc54258a6921ead194c1587" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.251089 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 871180863899419595cd632e7a4cf0dc (DEFAULT_TABLE table=default.Table_88abc71d734244088bac8733189eae55 [id=6f37f6d6318e45f1a016b8a34634050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.260645 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd370138ac4039be55642f1b4c5607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.260864 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871180863899419595cd632e7a4cf0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.266640 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f1d916a5ea48708c2c1a61c92fe137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.251386 1038 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd370138ac4039be55642f1b4c5607 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.270839 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd370138ac4039be55642f1b4c5607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.271355 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:12.271720 1184 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.271834 1184 ts_tablet_manager.cc:1397] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250110 02:51:12.272384 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.272552 1184 raft_consensus.cc:357] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.272747 1184 raft_consensus.cc:383] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.272821 1184 raft_consensus.cc:738] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.273026 1184 consensus_queue.cc:260] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.273519 1184 ts_tablet_manager.cc:1428] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:12.273806 1184 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.253597 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0588e5e5714cc5bd1c73665e5645b1 (DEFAULT_TABLE table=default.Table_ca9f3a9eeaa845498cf76982cc9b59fd [id=bda31fe0278a4b9a95eedb85d71b2eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.275135 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0588e5e5714cc5bd1c73665e5645b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.276063 1184 tablet_bootstrap.cc:654] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.253940 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 (DEFAULT_TABLE table=default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.278784 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f1d916a5ea48708c2c1a61c92fe137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.280385 1241 fs_manager.cc:730] Time spent opening block manager: real 0.142s user 0.003s sys 0.000s
I20250110 02:51:12.280578 1241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
uuid: "5d8d40556dc348e8aa8255d116251a10"
format_stamp: "Formatted at 2025-01-10 02:51:06 on dist-test-slave-bpn1"
I20250110 02:51:12.281785 1241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:12.254235 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (DEFAULT_TABLE table=default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.282651 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.286124 1184 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.286253 1184 ts_tablet_manager.cc:1397] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250110 02:51:12.287544 1184 raft_consensus.cc:357] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.287796 1184 raft_consensus.cc:383] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.288007 1184 raft_consensus.cc:738] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.288206 1184 consensus_queue.cc:260] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:12.290897 711 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.256001 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e3fc9826ef554a5699e365d10d02e97a (DEFAULT_TABLE table=default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.291864 1184 ts_tablet_manager.cc:1428] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20250110 02:51:12.292117 1184 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.292366 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.293179 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fc9826ef554a5699e365d10d02e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.293721 1184 tablet_bootstrap.cc:654] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.302172 1181 ts_tablet_manager.cc:1397] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.000s
I20250110 02:51:12.302944 1181 raft_consensus.cc:357] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.303134 1181 raft_consensus.cc:383] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.303205 1181 raft_consensus.cc:738] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.303416 1181 consensus_queue.cc:260] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.303815 888 tablet_service.cc:1467] Processing CreateTablet for tablet 871180863899419595cd632e7a4cf0dc (DEFAULT_TABLE table=default.Table_88abc71d734244088bac8733189eae55 [id=6f37f6d6318e45f1a016b8a34634050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.304246 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871180863899419595cd632e7a4cf0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.304556 1184 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.304737 1184 ts_tablet_manager.cc:1397] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20250110 02:51:12.305204 1181 ts_tablet_manager.cc:1428] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:12.305660 1178 raft_consensus.cc:2238] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.306138 1178 raft_consensus.cc:2267] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.306907 892 tablet_service.cc:1467] Processing CreateTablet for tablet d6fd370138ac4039be55642f1b4c5607 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.309470 1178 ts_tablet_manager.cc:1905] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.309592 898 tablet_service.cc:1467] Processing CreateTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (DEFAULT_TABLE table=default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.310031 886 tablet_service.cc:1467] Processing CreateTablet for tablet e3fc9826ef554a5699e365d10d02e97a (DEFAULT_TABLE table=default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.310286 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.310619 885 tablet_service.cc:1467] Processing CreateTablet for tablet b43bb790e85343308a1bb1e1110bfb9c (DEFAULT_TABLE table=default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4 [id=27a2d816626544e68abffc59b5980171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.310938 890 tablet_service.cc:1467] Processing CreateTablet for tablet 598bc151d0c14fc6bb28523c535d9366 (DEFAULT_TABLE table=default.Table_39db324aabe548a28775a241e962cfe7 [id=00766890e5f048ceb05d2732814df989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.311232 897 tablet_service.cc:1467] Processing CreateTablet for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (DEFAULT_TABLE table=default.Table_6123629172f54edb8b4da1696d019100 [id=1c37b52d30d44f5d9fbb15cca02d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.311549 887 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d6e0c043449688ed6450cb2a981a3 (DEFAULT_TABLE table=default.Table_b064f366436b4b72bc8428d0bbb0c3df [id=5b2f66225983487abf161e772efb6ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.312278 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2412420dc54258a6921ead194c1587" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.313850 1184 raft_consensus.cc:357] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.314065 1184 raft_consensus.cc:383] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.314179 1184 raft_consensus.cc:738] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.314558 1184 consensus_queue.cc:260] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.315263 1184 ts_tablet_manager.cc:1428] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:12.315701 1184 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:12.315726 709 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.315857 709 leader_election.cc:304] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.316570 1314 raft_consensus.cc:2743] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.317335 1179 ts_tablet_manager.cc:1918] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.317494 1179 log.cc:1198] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:12.318516 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd370138ac4039be55642f1b4c5607. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.318706 1179 ts_tablet_manager.cc:1939] T 2da6483de7f5475ea3d755a6efbda369 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.318742 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43bb790e85343308a1bb1e1110bfb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.319425 1184 tablet_bootstrap.cc:654] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.320209 895 tablet_service.cc:1467] Processing CreateTablet for tablet d20177229f7b4f11875e0007afb1dd00 (DEFAULT_TABLE table=default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.321763 881 tablet_service.cc:1467] Processing CreateTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 (DEFAULT_TABLE table=default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.323395 889 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (DEFAULT_TABLE table=default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.323637 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0588e5e5714cc5bd1c73665e5645b1 (DEFAULT_TABLE table=default.Table_ca9f3a9eeaa845498cf76982cc9b59fd [id=bda31fe0278a4b9a95eedb85d71b2eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.326444 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2da6483de7f5475ea3d755a6efbda369 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
I20250110 02:51:12.327134 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79175853474cbea951f5a5686bf630 (DEFAULT_TABLE table=default.Table_cccf34a241bd4aaf9296a023cdd9a853 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.328888 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 598bc151d0c14fc6bb28523c535d9366 (DEFAULT_TABLE table=default.Table_39db324aabe548a28775a241e962cfe7 [id=00766890e5f048ceb05d2732814df989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.329015 894 tablet_service.cc:1467] Processing CreateTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e (DEFAULT_TABLE table=default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.329504 1034 tablet_service.cc:1467] Processing CreateTablet for tablet d20177229f7b4f11875e0007afb1dd00 (DEFAULT_TABLE table=default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.330081 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d6e0c043449688ed6450cb2a981a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.330040 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (DEFAULT_TABLE table=default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.330689 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6d57f3e9a44d178300d3505d504854 (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.331236 1180 log.cc:1198] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/4c9d791b1c504171ba66dec55c8fb0e6
I20250110 02:51:12.332752 1180 ts_tablet_manager.cc:1939] T 4c9d791b1c504171ba66dec55c8fb0e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:12.334270 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 126) took 1486 ms. Trace:
I20250110 02:51:12.335253 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598bc151d0c14fc6bb28523c535d9366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.336063 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6d57f3e9a44d178300d3505d504854 (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.336345 1181 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:12.336642 1188 tablet_bootstrap.cc:492] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:12.337116 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f1d916a5ea48708c2c1a61c92fe137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.337299 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ddbc6b02742609bc2ca84c863a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.337380 1041 tablet_service.cc:1467] Processing CreateTablet for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (DEFAULT_TABLE table=default.Table_6123629172f54edb8b4da1696d019100 [id=1c37b52d30d44f5d9fbb15cca02d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.337857 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79175853474cbea951f5a5686bf630 (DEFAULT_TABLE table=default.Table_cccf34a241bd4aaf9296a023cdd9a853 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.338268 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e (DEFAULT_TABLE table=default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.338387 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d6e0c043449688ed6450cb2a981a3 (DEFAULT_TABLE table=default.Table_b064f366436b4b72bc8428d0bbb0c3df [id=5b2f66225983487abf161e772efb6ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.338711 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598bc151d0c14fc6bb28523c535d9366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.339215 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e79175853474cbea951f5a5686bf630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.340528 1181 tablet_bootstrap.cc:654] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.341583 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6d57f3e9a44d178300d3505d504854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.342523 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e88d1bbc304b4ebbc849a7d783052e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.338327 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b43bb790e85343308a1bb1e1110bfb9c (DEFAULT_TABLE table=default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4 [id=27a2d816626544e68abffc59b5980171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.342794 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ddbc6b02742609bc2ca84c863a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.343205 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa40c9c3f0ef447294b7c1c5f29ba06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.344343 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43bb790e85343308a1bb1e1110bfb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.346109 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d6e0c043449688ed6450cb2a981a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.346604 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20177229f7b4f11875e0007afb1dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.350234 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_eb30b4d9c27e4b5788054f7205e5610a" } modify_external_catalogs: true
I20250110 02:51:12.350476 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_eb30b4d9c27e4b5788054f7205e5610a" } modify_external_catalogs: true
I20250110 02:51:12.334362 1180 rpcz_store.cc:276] 0110 02:51:10.846989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.847043 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:12.211910 (+1364867us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d791b1c504171ba66dec55c8fb0e6
0110 02:51:12.211918 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.232407 (+ 20489us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.246269 (+ 13862us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.333442 (+ 87173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364422}
I20250110 02:51:12.352247 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20177229f7b4f11875e0007afb1dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.352530 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2d8cf27cea8e414ebe825a6b96c144c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.353068 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.354261 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d4422eca5604d149e73c2a669d07af0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.354430 1180 tablet_replica.cc:331] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:12.354621 1180 raft_consensus.cc:2238] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.356981 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa40c9c3f0ef447294b7c1c5f29ba06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.359668 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.360384 740 tablet_service.cc:1467] Processing CreateTablet for tablet b43bb790e85343308a1bb1e1110bfb9c (DEFAULT_TABLE table=default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4 [id=27a2d816626544e68abffc59b5980171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.360563 748 tablet_service.cc:1467] Processing CreateTablet for tablet 598bc151d0c14fc6bb28523c535d9366 (DEFAULT_TABLE table=default.Table_39db324aabe548a28775a241e962cfe7 [id=00766890e5f048ceb05d2732814df989]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.360831 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0588e5e5714cc5bd1c73665e5645b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.361096 743 tablet_service.cc:1467] Processing CreateTablet for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (DEFAULT_TABLE table=default.Table_6123629172f54edb8b4da1696d019100 [id=1c37b52d30d44f5d9fbb15cca02d70b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.362418 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b735b26f52fc4d4d9a3f706c984d008e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.362699 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fc9826ef554a5699e365d10d02e97a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:12.362980 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b735b26f52fc4d4d9a3f706c984d008e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.364531 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e79175853474cbea951f5a5686bf630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.364648 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:12.365442 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e88d1bbc304b4ebbc849a7d783052e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.365631 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6ffc0117db5547039d28e581cd91678b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.366101 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ffc0117db5547039d28e581cd91678b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.366189 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6d57f3e9a44d178300d3505d504854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.365579 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9cdcc4b295e54e45a928a008a61f0285"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.367470 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cdcc4b295e54e45a928a008a61f0285 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.370682 1188 ts_tablet_manager.cc:1397] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.000s
I20250110 02:51:12.371026 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.371548 1188 raft_consensus.cc:357] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.372262 1188 raft_consensus.cc:383] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.372160 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3832746ca2454627a4b2495825f5223c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.372246 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.372349 1188 raft_consensus.cc:738] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.372586 1188 consensus_queue.cc:260] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:12.372673 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3832746ca2454627a4b2495825f5223c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.373227 1188 ts_tablet_manager.cc:1428] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:12.373584 1188 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:12.374243 1314 raft_consensus.cc:491] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.374397 1314 raft_consensus.cc:513] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.375072 1314 leader_election.cc:290] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.375227 1314 raft_consensus.cc:491] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.375348 1314 raft_consensus.cc:513] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.375859 1314 leader_election.cc:290] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.375873 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_99c8d182fc924995a8f6e78bd31f07a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.376505 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99c8d182fc924995a8f6e78bd31f07a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:12.372674 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.383805 1184 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.383963 1184 ts_tablet_manager.cc:1397] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.001s
I20250110 02:51:12.384804 1184 raft_consensus.cc:357] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.385046 1184 raft_consensus.cc:383] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.385130 1184 raft_consensus.cc:738] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.385356 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_64d72c69126e4192abb401ad9e536c0a" } modify_external_catalogs: true
I20250110 02:51:12.385380 1184 consensus_queue.cc:260] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.385632 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_64d72c69126e4192abb401ad9e536c0a" } modify_external_catalogs: true
I20250110 02:51:12.385772 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5bfa749055ab45479cf96da68e537ece" } modify_external_catalogs: true
I20250110 02:51:12.385982 1184 ts_tablet_manager.cc:1428] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:12.385968 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5bfa749055ab45479cf96da68e537ece" } modify_external_catalogs: true
I20250110 02:51:12.386274 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_28491d38435d448b9ecbb83b5efb94d4" } new_table_name: "default.Table_055d44655c2743f3907923f2f49a33af" modify_external_catalogs: true
I20250110 02:51:12.390426 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 1602ade49b7442d6b8e6f4b85d2e80a1 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.391582 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:12.392275 518 catalog_manager.cc:6596] ReplaceTablet: tablet 7d4422eca5604d149e73c2a669d07af0 deleted and replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:12.392695 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_43eef7c99fbb4bffb1dc9eb5129cebae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.393167 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43eef7c99fbb4bffb1dc9eb5129cebae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.395290 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa40d20288644b2a8d36ede967d16e79" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.396052 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da6483de7f5475ea3d755a6efbda369 with delete_type TABLET_DATA_DELETED (replaced by tablet ad9f920cde954f31acebc6d0b6965e58) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.397980 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ee79c2aff75d4520b5ba281bc932d9d7" } new_table_name: "default.Table_2cb0f90ebf7248e1a9f43302db530a34" modify_external_catalogs: true
I20250110 02:51:12.405963 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_be1a53ef30a94d35ad3de81d4e9effee" } new_table_name: "default.Table_c12448a050c84ad3aa468fe82cc4337b" modify_external_catalogs: true
I20250110 02:51:12.419719 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 4c9d791b1c504171ba66dec55c8fb0e6 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:12.420377 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 89f6797b1076468a93fb0b510c553bed from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:12.421640 511 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8c3ce95413df40369ee599176f91b892 (table default.Table_a95ba565dbf940f79ed6f8b2eff1e74b [id=0743ba60cabd49c7a2c5230109d764eb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8c3ce95413df40369ee599176f91b892 consensus state has no leader
I20250110 02:51:12.422323 744 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d6e0c043449688ed6450cb2a981a3 (DEFAULT_TABLE table=default.Table_b064f366436b4b72bc8428d0bbb0c3df [id=5b2f66225983487abf161e772efb6ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.427125 749 tablet_service.cc:1467] Processing CreateTablet for tablet d20177229f7b4f11875e0007afb1dd00 (DEFAULT_TABLE table=default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.433781 1425 raft_consensus.cc:491] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.433961 1425 raft_consensus.cc:513] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.434540 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (DEFAULT_TABLE table=default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.435267 746 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79175853474cbea951f5a5686bf630 (DEFAULT_TABLE table=default.Table_cccf34a241bd4aaf9296a023cdd9a853 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.435283 1425 leader_election.cc:290] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.435870 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6d57f3e9a44d178300d3505d504854 (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.435923 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.436293 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.436256 745 tablet_service.cc:1467] Processing CreateTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e (DEFAULT_TABLE table=default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.436643 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.438846 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_36f7f022a56c43f5a9f52c6d2acbe5c4" } new_table_name: "default.Table_83b75cdbc7424f579e0297ecd7b0d4a4" modify_external_catalogs: true
I20250110 02:51:12.439621 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet e3fc9826ef554a5699e365d10d02e97a
W20250110 02:51:12.439821 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0127de45264f4660b843734a530eb310 consensus state has no leader
W20250110 02:51:12.440068 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f229f546a8d4a09b93f8b46c893d14a consensus state has no leader
I20250110 02:51:12.440479 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_526de8fc3ad6495ea9ec7e55e2accd58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.440963 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_526de8fc3ad6495ea9ec7e55e2accd58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.441031 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet dad0d561a8ef498ba13bc813d4cd82fa
W20250110 02:51:12.442693 709 leader_election.cc:343] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.443150 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.443574 752 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.444139 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b397397d667841eda2b5a948ca4a5015" } new_table_name: "default.Table_08d86300a7254466a9bff50b3f333765" modify_external_catalogs: true
I20250110 02:51:12.445326 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_196d49d7fc844b019f3a7356659f5b72" } modify_external_catalogs: true
I20250110 02:51:12.445415 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.439822 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8b2297377bc4e5095826e30523eede7 consensus state has no leader
W20250110 02:51:12.446051 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.446471 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423c820b81b845719d59a376c18e175a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.445541 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_196d49d7fc844b019f3a7356659f5b72" } modify_external_catalogs: true
I20250110 02:51:12.450196 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3f60c819f4824bc99d755e7442976" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.450728 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa40d20288644b2a8d36ede967d16e79" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.451370 1180 raft_consensus.cc:2267] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:12.452111 709 leader_election.cc:343] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:12.452623 709 leader_election.cc:343] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.454478 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.455296 879 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.456153 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.456581 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.460659 1180 ts_tablet_manager.cc:1905] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.439590 1179 tablet_replica.cc:331] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:12.462869 1179 raft_consensus.cc:2238] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.463899 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423c820b81b845719d59a376c18e175a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.470924 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3f60c819f4824bc99d755e7442976" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.471117 1061 raft_consensus.cc:2463] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:12.471966 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.473253 1188 tablet_bootstrap.cc:654] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.473603 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.473628 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.475013 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.494100 1178 ts_tablet_manager.cc:1918] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:12.495251 1178 log.cc:1198] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1aafff2e0de548129d99e9f07dc3ab11
I20250110 02:51:12.496811 1178 ts_tablet_manager.cc:1939] T 1aafff2e0de548129d99e9f07dc3ab11 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:12.502315 1241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:12.503960 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:12.504268 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:12.512543 1180 ts_tablet_manager.cc:1918] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.513098 1180 log.cc:1198] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a9f404a2e7494645af449aea3d3d3bf0
I20250110 02:51:12.517380 1180 ts_tablet_manager.cc:1939] T a9f404a2e7494645af449aea3d3d3bf0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:12.521173 1367 raft_consensus.cc:491] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.522203 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cea75045b3d94a008b85e2f3bbefdebc" } modify_external_catalogs: true
I20250110 02:51:12.522437 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cea75045b3d94a008b85e2f3bbefdebc" } modify_external_catalogs: true
I20250110 02:51:12.521365 1367 raft_consensus.cc:513] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.527149 1367 leader_election.cc:290] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:12.537482 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 131) took 1658 ms. Trace:
I20250110 02:51:12.538584 1178 rpcz_store.cc:276] 0110 02:51:10.878723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.878935 (+ 212us) service_pool.cc:225] Handling call
0110 02:51:12.241071 (+1362136us) ts_tablet_manager.cc:1145] Deleting tablet 1aafff2e0de548129d99e9f07dc3ab11
0110 02:51:12.241083 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.305989 (+ 64906us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x6160000c3454. stack: 00007ff9bee8efd6 00007ff9bee8ed90 00007ff9bde6e48d 000000000096d544 000000000096d50d 000000000096ceda 00007ff9ca731ed0 00007ff9cba26463 00007ff9ce4caeb7 00007ff9ce4fd584 00007ff9ce4de690 00007ff9ce4de41d 0000000000a71ecc 00007ff9beed5b64 00007ff9beed918b 00007ff9beed9008
0110 02:51:12.381267 (+ 75278us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.494014 (+112747us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.537150 (+ 43136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142834944,"tablet-delete.queue_time_us":1361606}
I20250110 02:51:12.540217 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1aafff2e0de548129d99e9f07dc3ab11 (table default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]) successfully deleted
I20250110 02:51:12.541612 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 136) took 1622 ms. Trace:
I20250110 02:51:12.542284 1178 rpcz_store.cc:276] 0110 02:51:10.918798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.919293 (+ 495us) service_pool.cc:225] Handling call
0110 02:51:12.540947 (+1621654us) ts_tablet_manager.cc:1145] Deleting tablet 0b74812cb81c44fcb8786cc5dae328e3
0110 02:51:12.540961 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.541461 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620894}
W20250110 02:51:12.543824 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 0b74812cb81c44fcb8786cc5dae328e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b74812cb81c44fcb8786cc5dae328e3 already in progress: creating tablet
W20250110 02:51:12.546247 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 585320c0970c4097b195caaae33789b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585320c0970c4097b195caaae33789b5 already in progress: creating tablet
W20250110 02:51:12.548182 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b already in progress: creating tablet
W20250110 02:51:12.550078 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9e72ed2d31a8467f8c092b48d593c1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e72ed2d31a8467f8c092b48d593c1cc already in progress: creating tablet
W20250110 02:51:12.552085 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8940c201aa104b0b8bc8d866859fe8ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8940c201aa104b0b8bc8d866859fe8ab already in progress: creating tablet
W20250110 02:51:12.553774 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5af3d61366b2432db53b18f2b7bb49d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5af3d61366b2432db53b18f2b7bb49d9 already in progress: creating tablet
I20250110 02:51:12.555986 1188 tablet_bootstrap.cc:492] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:12.556232 1188 ts_tablet_manager.cc:1397] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.183s user 0.004s sys 0.004s
I20250110 02:51:12.557250 1188 raft_consensus.cc:357] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.557601 1188 raft_consensus.cc:383] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.464066 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43bb790e85343308a1bb1e1110bfb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.464931 1184 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.557793 1188 raft_consensus.cc:738] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:12.472430 711 leader_election.cc:343] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.557963 711 leader_election.cc:304] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.558071 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e88d1bbc304b4ebbc849a7d783052e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:12.558346 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 85910us
W20250110 02:51:12.558837 711 leader_election.cc:343] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.559011 711 leader_election.cc:304] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.559451 711 leader_election.cc:304] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:12.526674 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:12.560279 1184 tablet_bootstrap.cc:654] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.560379 1314 raft_consensus.cc:2743] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.560889 1314 raft_consensus.cc:2743] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.561067 1314 raft_consensus.cc:2798] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:12.561142 1314 raft_consensus.cc:491] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:12.561234 1314 raft_consensus.cc:3054] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.562577 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20177229f7b4f11875e0007afb1dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.564685 1314 raft_consensus.cc:513] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.565987 1314 leader_election.cc:290] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.567181 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598bc151d0c14fc6bb28523c535d9366. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.567556 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3f60c819f4824bc99d755e7442976" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:12.568260 709 leader_election.cc:343] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.569706 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e79175853474cbea951f5a5686bf630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.570075 1188 consensus_queue.cc:260] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.570305 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa40c9c3f0ef447294b7c1c5f29ba06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.572525 1179 raft_consensus.cc:2267] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.572536 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d62161a3f0a2401198ea6728f350a9f6" } modify_external_catalogs: true
I20250110 02:51:12.572810 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d62161a3f0a2401198ea6728f350a9f6" } modify_external_catalogs: true
I20250110 02:51:12.573993 1179 ts_tablet_manager.cc:1905] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.574937 1188 ts_tablet_manager.cc:1428] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250110 02:51:12.555146 1178 tablet_replica.cc:331] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:12.577665 1178 raft_consensus.cc:2238] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.581048 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_28eb81570b2a4c609ee26c91c6aaf95e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.581672 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28eb81570b2a4c609ee26c91c6aaf95e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.583526 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6d57f3e9a44d178300d3505d504854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.584510 1178 raft_consensus.cc:2267] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.584766 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3d6e0c043449688ed6450cb2a981a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.585435 1184 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.585605 1184 ts_tablet_manager.cc:1397] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.121s user 0.008s sys 0.004s
I20250110 02:51:12.590042 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edfa33e30042edad853217aa761bd5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.590473 1184 raft_consensus.cc:357] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.590718 1184 raft_consensus.cc:383] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.592463 1184 raft_consensus.cc:738] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.592772 1184 consensus_queue.cc:260] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.593122 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d5ddbc6b02742609bc2ca84c863a9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.595106 853 leader_election.cc:304] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.595703 1367 raft_consensus.cc:2743] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.583868 1188 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:12.598798 1184 ts_tablet_manager.cc:1428] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
W20250110 02:51:12.603659 996 consensus_peers.cc:487] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:12.603801 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3f60c819f4824bc99d755e7442976" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:12.605567 916 raft_consensus.cc:3054] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.607003 1184 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.609051 1060 raft_consensus.cc:3054] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.615898 1314 raft_consensus.cc:491] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.619072 1178 ts_tablet_manager.cc:1905] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.620836 1314 raft_consensus.cc:513] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.623039 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 128) took 1757 ms. Trace:
I20250110 02:51:12.625034 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.636513 1188 tablet_bootstrap.cc:654] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.642242 1178 ts_tablet_manager.cc:1918] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 02:51:12.643101 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b43bb790e85343308a1bb1e1110bfb9c consensus state has no leader
I20250110 02:51:12.649647 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_108282f5803c488098a01d20b1e4f31a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.650152 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_108282f5803c488098a01d20b1e4f31a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.650192 1178 log.cc:1198] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/a9f404a2e7494645af449aea3d3d3bf0
I20250110 02:51:12.651440 1178 ts_tablet_manager.cc:1939] T a9f404a2e7494645af449aea3d3d3bf0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:12.651846 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 163) took 1017 ms. Trace:
I20250110 02:51:12.651993 1178 rpcz_store.cc:276] 0110 02:51:11.634843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.634904 (+ 61us) service_pool.cc:225] Handling call
0110 02:51:12.554559 (+919655us) ts_tablet_manager.cc:1145] Deleting tablet a9f404a2e7494645af449aea3d3d3bf0
0110 02:51:12.554728 (+ 169us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.638035 (+ 83307us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.642141 (+ 4106us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.651818 (+ 9677us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43,"tablet-delete.queue_time_us":811646}
I20250110 02:51:12.652050 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.652415 1178 tablet_replica.cc:331] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:12.652515 1180 rpcz_store.cc:276] 0110 02:51:10.847224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.847523 (+ 299us) service_pool.cc:225] Handling call
0110 02:51:12.352182 (+1504659us) ts_tablet_manager.cc:1145] Deleting tablet a9f404a2e7494645af449aea3d3d3bf0
0110 02:51:12.352192 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.497846 (+145654us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.512415 (+ 14569us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.604617 (+ 92202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504410}
I20250110 02:51:12.652881 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a9f404a2e7494645af449aea3d3d3bf0 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:12.652953 1180 tablet_replica.cc:331] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:12.652634 1178 raft_consensus.cc:2238] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.653220 1178 raft_consensus.cc:2267] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.653460 1180 raft_consensus.cc:2238] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:12.653803 1180 raft_consensus.cc:2267] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.654856 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet a9f404a2e7494645af449aea3d3d3bf0 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:12.655037 1314 leader_election.cc:290] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:12.655576 709 leader_election.cc:343] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.656502 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.657022 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.657609 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4de67338a26f4febb5dd7611a72b4723 (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.657789 1181 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:12.657938 1181 ts_tablet_manager.cc:1397] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.352s user 0.061s sys 0.140s
I20250110 02:51:12.658032 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de67338a26f4febb5dd7611a72b4723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.658321 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_23df132302f5414db477144cbb57b923"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.658581 711 leader_election.cc:343] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.658694 711 leader_election.cc:304] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:12.658877 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23df132302f5414db477144cbb57b923 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.660017 1181 raft_consensus.cc:357] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.660288 1181 raft_consensus.cc:383] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.660506 1181 raft_consensus.cc:738] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.660738 1181 consensus_queue.cc:260] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.662122 1446 tablet.cc:1721] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:12.662608 1181 ts_tablet_manager.cc:1428] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:12.663197 1178 ts_tablet_manager.cc:1905] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:12.664165 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf711522866d46439d7ee69aa63d7f39 consensus state has no leader
I20250110 02:51:12.664542 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c36e12f8b3e64b41980128408d36846f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.665041 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c36e12f8b3e64b41980128408d36846f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.665400 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 4de67338a26f4febb5dd7611a72b4723 (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.663798 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ab917bde534e3fbdf5dbae4af9304e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.671110 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b7d4e47c779b4a43a9e1719d181bf639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.671522 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7d4e47c779b4a43a9e1719d181bf639 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.674783 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1602ade49b7442d6b8e6f4b85d2e80a1
I20250110 02:51:12.675204 525 catalog_manager.cc:6596] ReplaceTablet: tablet 1602ade49b7442d6b8e6f4b85d2e80a1 deleted and replaced by tablet 302af84fd8374ffa96a7503b54849536
I20250110 02:51:12.675511 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5c647fe509fd45eebc5796422650b5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.675961 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c647fe509fd45eebc5796422650b5d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.679893 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f4778347f054b86af697dbe3b287263 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.681947 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:12.682864 524 master_service.cc:946] ReplaceTablet: received request to replace tablet 5af3d61366b2432db53b18f2b7bb49d9 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.683375 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7489371158674f9abe9b0cafbd4f1157" } new_table_name: "default.Table_3f8f54a3c0194e228523d98131042d67" modify_external_catalogs: true
I20250110 02:51:12.688611 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:12.689218 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6eb43dd09ca247e99abdb0335957a77f" } new_table_name: "default.Table_8df25635dd7e4d58818dc14fc678e221" modify_external_catalogs: true
I20250110 02:51:12.690409 1188 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:12.690840 1427 raft_consensus.cc:491] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.690992 1427 raft_consensus.cc:513] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.691764 1427 leader_election.cc:290] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.690569 1188 ts_tablet_manager.cc:1397] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.001s
I20250110 02:51:12.690321 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:12.692729 1188 raft_consensus.cc:357] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.692952 1188 raft_consensus.cc:383] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.693024 1188 raft_consensus.cc:738] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.693231 1188 consensus_queue.cc:260] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.694579 1314 raft_consensus.cc:2743] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.694628 1184 tablet_bootstrap.cc:654] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.695535 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d133af1dd7834a27b1d3ee0e4e788fe3" } modify_external_catalogs: true
I20250110 02:51:12.695745 1188 ts_tablet_manager.cc:1428] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250110 02:51:12.695976 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d133af1dd7834a27b1d3ee0e4e788fe3" } modify_external_catalogs: true
I20250110 02:51:12.697085 1188 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:12.697911 522 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:12.697974 1179 ts_tablet_manager.cc:1918] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.698081 1179 log.cc:1198] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/67adef4c3f4041e1ab480100ca762f62
I20250110 02:51:12.698881 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_309dd7a424574e50a1d844d112efa307" } modify_external_catalogs: true
I20250110 02:51:12.699052 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_309dd7a424574e50a1d844d112efa307" } modify_external_catalogs: true
I20250110 02:51:12.694286 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c8a984ce6fa04f7687f873fff9a693d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.700397 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8a984ce6fa04f7687f873fff9a693d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.700501 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.701236 1188 tablet_bootstrap.cc:654] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.701747 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 302af84fd8374ffa96a7503b54849536) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.702265 748 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.702422 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.705871 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.708799 526 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:12.708890 1184 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.709033 1184 ts_tablet_manager.cc:1397] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.000s
I20250110 02:51:12.709684 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6123629172f54edb8b4da1696d019100" } new_table_name: "default.Table_f35eeea899a4401f8670ce65182c2f79" modify_external_catalogs: true
I20250110 02:51:12.709810 1184 raft_consensus.cc:357] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.710081 1184 raft_consensus.cc:383] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.710144 1179 ts_tablet_manager.cc:1939] T 67adef4c3f4041e1ab480100ca762f62 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.710171 1184 raft_consensus.cc:738] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.710767 1184 consensus_queue.cc:260] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.710919 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 173) took 1014 ms. Trace:
I20250110 02:51:12.711163 520 master_service.cc:946] ReplaceTablet: received request to replace tablet d20177229f7b4f11875e0007afb1dd00 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.711292 1184 ts_tablet_manager.cc:1428] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:12.711019 1179 rpcz_store.cc:276] 0110 02:51:11.696485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.696550 (+ 65us) service_pool.cc:225] Handling call
0110 02:51:12.319570 (+623020us) ts_tablet_manager.cc:1145] Deleting tablet 67adef4c3f4041e1ab480100ca762f62
0110 02:51:12.319580 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.688473 (+368893us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.697845 (+ 9372us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.710876 (+ 13031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":622767}
I20250110 02:51:12.711597 1184 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.711992 1179 tablet_replica.cc:331] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:12.712101 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 67adef4c3f4041e1ab480100ca762f62 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
I20250110 02:51:12.712319 1179 raft_consensus.cc:2238] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:12.714116 994 consensus_peers.cc:487] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:12.714726 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8f4778347f054b86af697dbe3b287263
I20250110 02:51:12.715864 521 catalog_manager.cc:6596] ReplaceTablet: tablet 8f4778347f054b86af697dbe3b287263 deleted and replaced by tablet abbd43638a2c47dcba4f6f90e8676f42
I20250110 02:51:12.712666 1179 raft_consensus.cc:2267] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.713029 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4de67338a26f4febb5dd7611a72b4723 (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.718684 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.718967 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de67338a26f4febb5dd7611a72b4723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.719532 1184 tablet_bootstrap.cc:654] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.722404 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b6d57f3e9a44d178300d3505d504854 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.722431 1179 ts_tablet_manager.cc:1905] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.722965 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514704cd935b44e8b760221224332509" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.723107 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 302af84fd8374ffa96a7503b54849536) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.723522 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.723902 895 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.723924 1060 raft_consensus.cc:2463] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:12.724184 1181 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:12.724259 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.724524 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:12.725355 709 leader_election.cc:343] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.725672 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de67338a26f4febb5dd7611a72b4723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.725669 1180 ts_tablet_manager.cc:1905] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.732681 1314 raft_consensus.cc:491] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.732868 1314 raft_consensus.cc:513] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.733481 1314 leader_election.cc:290] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.734310 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.734462 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 302af84fd8374ffa96a7503b54849536) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.734853 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:12.735436 709 leader_election.cc:343] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.738598 1179 ts_tablet_manager.cc:1918] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:12.738760 1179 log.cc:1198] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/308f307f4200409aaa47df55f20472d0
I20250110 02:51:12.739152 1179 ts_tablet_manager.cc:1939] T 308f307f4200409aaa47df55f20472d0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.742172 1184 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.742298 1184 ts_tablet_manager.cc:1397] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.007s
I20250110 02:51:12.744599 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.746042 711 leader_election.cc:304] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:12.746230 1184 raft_consensus.cc:357] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.746454 1184 raft_consensus.cc:383] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.746600 1314 raft_consensus.cc:2798] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:12.746763 1314 raft_consensus.cc:695] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:12.747037 1314 consensus_queue.cc:237] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.747494 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 174) took 1047 ms. Trace:
I20250110 02:51:12.747589 1179 rpcz_store.cc:276] 0110 02:51:11.699797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.699841 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:12.711931 (+1012090us) ts_tablet_manager.cc:1145] Deleting tablet 308f307f4200409aaa47df55f20472d0
0110 02:51:12.711938 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.735582 (+ 23644us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.738446 (+ 2864us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.747417 (+ 8971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011914}
I20250110 02:51:12.748013 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 175) took 1041 ms. Trace:
I20250110 02:51:12.748131 1184 raft_consensus.cc:738] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.748322 1184 consensus_queue.cc:260] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.748394 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
I20250110 02:51:12.748090 1179 rpcz_store.cc:276] 0110 02:51:11.707021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.707080 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:12.747856 (+1040776us) ts_tablet_manager.cc:1145] Deleting tablet 8940c201aa104b0b8bc8d866859fe8ab
0110 02:51:12.747865 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.747963 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040476}
I20250110 02:51:12.748934 1184 ts_tablet_manager.cc:1428] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250110 02:51:12.749166 1184 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.749395 1179 tablet_replica.cc:331] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:12.749598 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8940c201aa104b0b8bc8d866859fe8ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8940c201aa104b0b8bc8d866859fe8ab already in progress: creating tablet
W20250110 02:51:12.749826 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 019c0d60da5548bb86c93ccd89bf94ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019c0d60da5548bb86c93ccd89bf94ff already in progress: creating tablet
W20250110 02:51:12.750051 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f14f06ba7d0f44418693e36f6bc6d888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14f06ba7d0f44418693e36f6bc6d888 already in progress: creating tablet
I20250110 02:51:12.751009 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.751209 1179 raft_consensus.cc:2238] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:12.751324 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b91f351e02d406f8675eb44c4c04ced consensus state has no leader
I20250110 02:51:12.751334 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.752007 1179 raft_consensus.cc:2267] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.752523 1179 ts_tablet_manager.cc:1905] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:12.752732 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60e534bf66f044b88d8cda0f3a1228f4 consensus state has no leader
I20250110 02:51:12.755246 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514704cd935b44e8b760221224332509" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.755482 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f5e88d1bbc304b4ebbc849a7d783052e
W20250110 02:51:12.755847 711 leader_election.cc:343] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.755993 711 leader_election.cc:304] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.756929 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.757416 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:12.757652 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:12.758108 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa40c9c3f0ef447294b7c1c5f29ba06e consensus state has no leader
I20250110 02:51:12.759028 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.760135 1427 raft_consensus.cc:2743] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.760365 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:12.761039 711 leader_election.cc:343] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.761168 711 leader_election.cc:304] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.761395 1427 raft_consensus.cc:2743] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.762084 1179 ts_tablet_manager.cc:1918] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.762234 1179 log.cc:1198] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/0c2412420dc54258a6921ead194c1587
I20250110 02:51:12.762431 1181 tablet_bootstrap.cc:654] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.762703 1179 ts_tablet_manager.cc:1939] T 0c2412420dc54258a6921ead194c1587 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.763693 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 0c2412420dc54258a6921ead194c1587 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
I20250110 02:51:12.751227 1184 tablet_bootstrap.cc:654] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:12.764362 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 48cb20d47ace4fd49f9dc4e526cab702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48cb20d47ace4fd49f9dc4e526cab702 already in progress: creating tablet
I20250110 02:51:12.765440 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:12.767428 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 95b14d7592024a34baf7756d92e546d9 (DEFAULT_TABLE table=default.Table_43eef7c99fbb4bffb1dc9eb5129cebae [id=8dafaf90f6e7460284b6fc3434eed1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.767594 895 tablet_service.cc:1467] Processing CreateTablet for tablet 95b14d7592024a34baf7756d92e546d9 (DEFAULT_TABLE table=default.Table_43eef7c99fbb4bffb1dc9eb5129cebae [id=8dafaf90f6e7460284b6fc3434eed1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.767685 1179 tablet_replica.cc:331] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:12.767803 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b14d7592024a34baf7756d92e546d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.767880 1179 raft_consensus.cc:2238] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:12.767978 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b14d7592024a34baf7756d92e546d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.768044 1179 raft_consensus.cc:2267] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.768615 1179 ts_tablet_manager.cc:1905] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:12.769729 756 tablet_service.cc:1467] Processing CreateTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.770061 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76833ce515cf4ee9bc8fff0f626af220. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.770881 754 tablet_service.cc:1467] Processing CreateTablet for tablet 940445840db3441598eba57d3ccb8c9c (DEFAULT_TABLE table=default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.771199 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940445840db3441598eba57d3ccb8c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.771411 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 940445840db3441598eba57d3ccb8c9c (DEFAULT_TABLE table=default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.771739 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940445840db3441598eba57d3ccb8c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.772799 744 tablet_service.cc:1467] Processing CreateTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.773799 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.776880 745 tablet_service.cc:1467] Processing CreateTablet for tablet b375a932156d4abb908dbf327689b0f9 (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.777206 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b375a932156d4abb908dbf327689b0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.778177 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.778424 743 tablet_service.cc:1467] Processing CreateTablet for tablet 28340f0f9a2043a199c74f13755eb702 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.778800 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28340f0f9a2043a199c74f13755eb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.781962 887 tablet_service.cc:1467] Processing CreateTablet for tablet 940445840db3441598eba57d3ccb8c9c (DEFAULT_TABLE table=default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.782346 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 940445840db3441598eba57d3ccb8c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.782663 748 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f (DEFAULT_TABLE table=default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.782963 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd1e726f1a41d0894737b3ef3caf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.783577 1184 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.783751 1184 ts_tablet_manager.cc:1397] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20250110 02:51:12.784945 1184 raft_consensus.cc:357] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.785142 1184 raft_consensus.cc:383] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.785209 1184 raft_consensus.cc:738] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.785430 1184 consensus_queue.cc:260] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.786700 897 tablet_service.cc:1467] Processing CreateTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.786782 740 tablet_service.cc:1467] Processing CreateTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.787109 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76833ce515cf4ee9bc8fff0f626af220. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.787195 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.788225 1184 ts_tablet_manager.cc:1428] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:12.788506 1184 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.790035 1184 tablet_bootstrap.cc:654] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.790632 886 tablet_service.cc:1467] Processing CreateTablet for tablet 28340f0f9a2043a199c74f13755eb702 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.790977 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28340f0f9a2043a199c74f13755eb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.791509 749 tablet_service.cc:1467] Processing CreateTablet for tablet eff9b10c419242cf85ef1b708bd5a862 (DEFAULT_TABLE table=default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.791810 1188 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:12.791824 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff9b10c419242cf85ef1b708bd5a862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.791946 1188 ts_tablet_manager.cc:1397] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.016s
I20250110 02:51:12.794528 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76833ce515cf4ee9bc8fff0f626af220. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.794632 880 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f (DEFAULT_TABLE table=default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.794909 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd1e726f1a41d0894737b3ef3caf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.795235 1188 raft_consensus.cc:357] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.795449 1188 raft_consensus.cc:383] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.795568 1188 raft_consensus.cc:738] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.797425 1188 consensus_queue.cc:260] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.798054 1188 ts_tablet_manager.cc:1428] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:12.798130 881 tablet_service.cc:1467] Processing CreateTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.798314 1188 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:12.798508 898 tablet_service.cc:1467] Processing CreateTablet for tablet eff9b10c419242cf85ef1b708bd5a862 (DEFAULT_TABLE table=default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.798609 1034 tablet_service.cc:1467] Processing CreateTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.798874 882 tablet_service.cc:1467] Processing CreateTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.799870 1188 tablet_bootstrap.cc:654] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.800032 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f (DEFAULT_TABLE table=default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.800307 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 28340f0f9a2043a199c74f13755eb702 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.801419 1037 tablet_service.cc:1467] Processing CreateTablet for tablet eff9b10c419242cf85ef1b708bd5a862 (DEFAULT_TABLE table=default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.801942 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.804491 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff9b10c419242cf85ef1b708bd5a862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.805388 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28340f0f9a2043a199c74f13755eb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.806293 1184 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.806437 1184 ts_tablet_manager.cc:1397] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20250110 02:51:12.806619 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd1e726f1a41d0894737b3ef3caf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.807222 1184 raft_consensus.cc:357] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.807417 1184 raft_consensus.cc:383] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.807488 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.807508 1184 raft_consensus.cc:738] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.807730 1184 consensus_queue.cc:260] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.810657 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.810689 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff9b10c419242cf85ef1b708bd5a862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.813982 1386 raft_consensus.cc:491] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.816519 1386 raft_consensus.cc:513] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.817174 1386 leader_election.cc:290] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:12.817200 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:12.817620 521 catalog_manager.cc:6596] ReplaceTablet: tablet 3b6d57f3e9a44d178300d3505d504854 deleted and replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b
I20250110 02:51:12.818639 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da6483de7f5475ea3d755a6efbda369" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.818854 916 raft_consensus.cc:2463] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:12.818806 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (DEFAULT_TABLE table=default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.819146 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.820509 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:12.823266 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.823475 1184 ts_tablet_manager.cc:1428] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250110 02:51:12.823781 1184 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.824779 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b375a932156d4abb908dbf327689b0f9 (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.825351 1184 tablet_bootstrap.cc:654] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:12.825430 996 leader_election.cc:343] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:12.826208 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b375a932156d4abb908dbf327689b0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.827239 1179 ts_tablet_manager.cc:1918] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.827371 1179 log.cc:1198] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/64c77ec8c7724d5fb408a8bfa53811eb
I20250110 02:51:12.827436 889 tablet_service.cc:1467] Processing CreateTablet for tablet b375a932156d4abb908dbf327689b0f9 (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.827728 1179 ts_tablet_manager.cc:1939] T 64c77ec8c7724d5fb408a8bfa53811eb P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:12.827852 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b375a932156d4abb908dbf327689b0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.828639 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 64c77ec8c7724d5fb408a8bfa53811eb (table default.Table_e676f1148b5f4988a2be33981eec57c3 [id=d7bbf5b39b224708974b5f6da6429fe7]) successfully deleted
W20250110 02:51:12.829257 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9dbb61a51f14a3f96cc3c77981e8ce5 already in progress: creating tablet
I20250110 02:51:12.830170 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.831266 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.831712 994 leader_election.cc:304] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.832043 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:12.832214 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3ae68e4f0254ef38aafd5a1f8f88a54 already in progress: creating tablet
I20250110 02:51:12.832496 1179 tablet_replica.cc:331] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:12.833273 1386 raft_consensus.cc:2798] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:12.833376 1386 raft_consensus.cc:491] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:12.833756 1386 raft_consensus.cc:3054] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.834497 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.847246 1180 ts_tablet_manager.cc:1918] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:12.847676 1180 log.cc:1198] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1aafff2e0de548129d99e9f07dc3ab11
I20250110 02:51:12.848187 1180 ts_tablet_manager.cc:1939] T 1aafff2e0de548129d99e9f07dc3ab11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:12.848321 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f35eeea899a4401f8670ce65182c2f79" } new_table_name: "default.Table_7e7300d4fc9b49639885bbf4895cd07f" modify_external_catalogs: true
I20250110 02:51:12.848922 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 127) took 2001 ms. Trace:
I20250110 02:51:12.849035 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.850366 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.850834 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f15078fc22f4b32a48c7190b36e2192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.851878 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f15078fc22f4b32a48c7190b36e2192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.855715 1427 raft_consensus.cc:491] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.855877 1427 raft_consensus.cc:513] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:12.856421 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa40c9c3f0ef447294b7c1c5f29ba06e consensus state has no leader
I20250110 02:51:12.856573 1427 leader_election.cc:290] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.857486 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.857710 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.858433 746 tablet_service.cc:1467] Processing CreateTablet for tablet 95b14d7592024a34baf7756d92e546d9 (DEFAULT_TABLE table=default.Table_43eef7c99fbb4bffb1dc9eb5129cebae [id=8dafaf90f6e7460284b6fc3434eed1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.859059 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b74812cb81c44fcb8786cc5dae328e3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:12.859428 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b74812cb81c44fcb8786cc5dae328e3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:12.860038 711 leader_election.cc:343] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:12.860340 709 leader_election.cc:343] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:12.860458 709 leader_election.cc:304] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.860420 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_b0b62b5867624cd1b77150b7b3d2090d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.860715 1427 raft_consensus.cc:2743] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:12.860946 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0b62b5867624cd1b77150b7b3d2090d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.861335 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b14d7592024a34baf7756d92e546d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.866223 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_bc03ee1fa76e458c8029c902c813dcb2" } new_table_name: "default.Table_65e2869747e148ca99fe45b205855ef9" modify_external_catalogs: true
I20250110 02:51:12.866501 880 tablet_service.cc:1467] Processing CreateTablet for tablet 240b64d551a548e884ee2a7238c0363a (DEFAULT_TABLE table=default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.867290 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240b64d551a548e884ee2a7238c0363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.867774 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1390f867a3a8436092d1920daff4c5b0 (DEFAULT_TABLE table=default.Table_b7d4e47c779b4a43a9e1719d181bf639 [id=81d7d3112de44cae840ee0f1f86e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.868137 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 240b64d551a548e884ee2a7238c0363a (DEFAULT_TABLE table=default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.868795 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1390f867a3a8436092d1920daff4c5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.869007 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240b64d551a548e884ee2a7238c0363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.872402 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_009562f3b1ed4d928d6fb21a604b051b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.872610 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_fdcd90a5deda47968fc8ba979edbf5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.872844 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_009562f3b1ed4d928d6fb21a604b051b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:12.872962 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdcd90a5deda47968fc8ba979edbf5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.873809 895 tablet_service.cc:1467] Processing CreateTablet for tablet 1390f867a3a8436092d1920daff4c5b0 (DEFAULT_TABLE table=default.Table_b7d4e47c779b4a43a9e1719d181bf639 [id=81d7d3112de44cae840ee0f1f86e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.874140 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1390f867a3a8436092d1920daff4c5b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:12.874640 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da8f247c862e4cb68452e230857ee6b0 consensus state has no leader
I20250110 02:51:12.875088 1184 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.878129 898 tablet_service.cc:1467] Processing CreateTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.878938 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f15078fc22f4b32a48c7190b36e2192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.880677 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9507a5283d8f4d07a293ba474c642298 (DEFAULT_TABLE table=default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.881068 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9507a5283d8f4d07a293ba474c642298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.883744 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6ffc0117db5547039d28e581cd91678b" } modify_external_catalogs: true
I20250110 02:51:12.884058 1367 raft_consensus.cc:491] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.884167 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9507a5283d8f4d07a293ba474c642298 (DEFAULT_TABLE table=default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.884199 1367 raft_consensus.cc:513] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } }
I20250110 02:51:12.884846 1367 leader_election.cc:290] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:12.883903 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6ffc0117db5547039d28e581cd91678b" } modify_external_catalogs: true
W20250110 02:51:12.886890 853 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 4c9d791b1c504171ba66dec55c8fb0e6
I20250110 02:51:12.887452 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6e758fcd046f4196a3735d4b731544e6 (DEFAULT_TABLE table=default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a [id=90471593be8d4c04b0ac2d50a96966e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.888785 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1390f867a3a8436092d1920daff4c5b0 (DEFAULT_TABLE table=default.Table_b7d4e47c779b4a43a9e1719d181bf639 [id=81d7d3112de44cae840ee0f1f86e6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.888690 1184 ts_tablet_manager.cc:1397] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.012s
I20250110 02:51:12.889297 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9507a5283d8f4d07a293ba474c642298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.889644 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1390f867a3a8436092d1920daff4c5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.889950 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e758fcd046f4196a3735d4b731544e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.889694 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6e758fcd046f4196a3735d4b731544e6 (DEFAULT_TABLE table=default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a [id=90471593be8d4c04b0ac2d50a96966e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.890992 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7ab0055319bd4962a0267ce290806e0f" } modify_external_catalogs: true
I20250110 02:51:12.891160 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7ab0055319bd4962a0267ce290806e0f" } modify_external_catalogs: true
I20250110 02:51:12.891211 886 tablet_service.cc:1467] Processing CreateTablet for tablet fb2272dfd0b14113afa3e13b38de6f8c (DEFAULT_TABLE table=default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 [id=9643d639878f4122bf42aeca200f9c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.891499 754 tablet_service.cc:1467] Processing CreateTablet for tablet 240b64d551a548e884ee2a7238c0363a (DEFAULT_TABLE table=default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.891917 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 240b64d551a548e884ee2a7238c0363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.849020 1180 rpcz_store.cc:276] 0110 02:51:10.847107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.851454 (+ 4347us) service_pool.cc:225] Handling call
0110 02:51:12.652882 (+1801428us) ts_tablet_manager.cc:1145] Deleting tablet 1aafff2e0de548129d99e9f07dc3ab11
0110 02:51:12.652893 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.760128 (+107235us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.831568 (+ 71440us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.848856 (+ 17288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800734}
I20250110 02:51:12.895470 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e758fcd046f4196a3735d4b731544e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.895500 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_61e4e1bf02224dce873e1aa1bf273c54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.895619 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 9507a5283d8f4d07a293ba474c642298 (DEFAULT_TABLE table=default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:12.896026 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61e4e1bf02224dce873e1aa1bf273c54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.897799 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e55e81b07bfe4a668255066cb6447735" } modify_external_catalogs: true
I20250110 02:51:12.897954 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e55e81b07bfe4a668255066cb6447735" } modify_external_catalogs: true
I20250110 02:51:12.899415 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2d364ce704a5437e83b3001938e425cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.899896 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d364ce704a5437e83b3001938e425cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:12.900681 853 leader_election.cc:343] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 4c9d791b1c504171ba66dec55c8fb0e6
I20250110 02:51:12.900816 853 leader_election.cc:304] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.901501 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1aafff2e0de548129d99e9f07dc3ab11 (table default.Table_1eb291591c244ffa92efc9d213a760b9 [id=5cfa22d81ad140c5b286f93627039499]) successfully deleted
I20250110 02:51:12.901551 1367 raft_consensus.cc:2743] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.895992 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9507a5283d8f4d07a293ba474c642298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.904946 1184 raft_consensus.cc:357] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.905165 1184 raft_consensus.cc:383] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.905228 1184 raft_consensus.cc:738] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.905457 1184 consensus_queue.cc:260] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.906564 1179 raft_consensus.cc:2238] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:12.906760 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:12.908084 749 tablet_service.cc:1467] Processing CreateTablet for tablet fb2272dfd0b14113afa3e13b38de6f8c (DEFAULT_TABLE table=default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 [id=9643d639878f4122bf42aeca200f9c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.908389 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2272dfd0b14113afa3e13b38de6f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.908941 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:12.910619 744 tablet_service.cc:1467] Processing CreateTablet for tablet abbd43638a2c47dcba4f6f90e8676f42 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.910699 1386 raft_consensus.cc:513] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.910965 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbd43638a2c47dcba4f6f90e8676f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.912133 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da6483de7f5475ea3d755a6efbda369" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:12.912333 916 raft_consensus.cc:3054] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.913046 882 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.913291 897 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.913635 889 tablet_service.cc:1467] Processing CreateTablet for tablet abbd43638a2c47dcba4f6f90e8676f42 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.913991 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbd43638a2c47dcba4f6f90e8676f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.914484 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.918605 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:12.919350 916 raft_consensus.cc:2463] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:12.920379 526 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:12.920810 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.920879 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3510e86314144a2ca012aacf3d78eacc (DEFAULT_TABLE table=default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.921175 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3510e86314144a2ca012aacf3d78eacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.924660 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.925194 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.925398 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 89f6797b1076468a93fb0b510c553bed from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.926319 749 tablet_service.cc:1467] Processing CreateTablet for tablet 971f97a3be9b49958b1218a275e126d4 (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.926733 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971f97a3be9b49958b1218a275e126d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.927032 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ecf7ad441f824cbe9e7384fb02a46754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:12.927333 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e7d7ae7a9eea4238802be676c6f59905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.927493 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecf7ad441f824cbe9e7384fb02a46754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:12.927811 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d7ae7a9eea4238802be676c6f59905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.930197 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_053712a8fc234f22a1cf1caf96485c71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.930757 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_053712a8fc234f22a1cf1caf96485c71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.931838 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 971f97a3be9b49958b1218a275e126d4 (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.932602 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.932739 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.933645 890 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4d2bb0b992474d8e5d21abaca5411c (DEFAULT_TABLE table=default.Table_28eb81570b2a4c609ee26c91c6aaf95e [id=b8fb31b88bef421ca915a74593e95dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.934039 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4d2bb0b992474d8e5d21abaca5411c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.936228 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4d2bb0b992474d8e5d21abaca5411c (DEFAULT_TABLE table=default.Table_28eb81570b2a4c609ee26c91c6aaf95e [id=b8fb31b88bef421ca915a74593e95dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.936568 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4d2bb0b992474d8e5d21abaca5411c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:12.940268 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 0127de45264f4660b843734a530eb310 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.940786 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3510e86314144a2ca012aacf3d78eacc (DEFAULT_TABLE table=default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.941061 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6e758fcd046f4196a3735d4b731544e6 (DEFAULT_TABLE table=default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a [id=90471593be8d4c04b0ac2d50a96966e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.941320 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.941490 897 tablet_service.cc:1467] Processing CreateTablet for tablet 971f97a3be9b49958b1218a275e126d4 (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:12.941560 996 leader_election.cc:343] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:12.945703 1179 pending_rounds.cc:62] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:12.945784 1179 pending_rounds.cc:69] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611728374120448 op_type: NO_OP noop_request { }
I20250110 02:51:12.945935 1179 raft_consensus.cc:2883] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:12.946045 1179 raft_consensus.cc:2267] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:12.946364 994 leader_election.cc:304] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.943764 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4d2bb0b992474d8e5d21abaca5411c (DEFAULT_TABLE table=default.Table_28eb81570b2a4c609ee26c91c6aaf95e [id=b8fb31b88bef421ca915a74593e95dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.949398 1178 ts_tablet_manager.cc:1918] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:12.949532 1178 log.cc:1198] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/67adef4c3f4041e1ab480100ca762f62
I20250110 02:51:12.949848 1178 ts_tablet_manager.cc:1939] T 67adef4c3f4041e1ab480100ca762f62 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:12.950176 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 166) took 1250 ms. Trace:
I20250110 02:51:12.950256 1178 rpcz_store.cc:276] 0110 02:51:11.699464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.699529 (+ 65us) service_pool.cc:225] Handling call
0110 02:51:12.652344 (+952815us) ts_tablet_manager.cc:1145] Deleting tablet 67adef4c3f4041e1ab480100ca762f62
0110 02:51:12.652351 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.737435 (+ 85084us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.949305 (+211870us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:12.950132 (+ 827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":907382}
I20250110 02:51:12.950692 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 67adef4c3f4041e1ab480100ca762f62 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
I20250110 02:51:12.951977 525 master_service.cc:946] ReplaceTablet: received request to replace tablet dad0d561a8ef498ba13bc813d4cd82fa from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:12.952538 1184 ts_tablet_manager.cc:1428] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250110 02:51:12.953117 1184 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.955330 1386 leader_election.cc:290] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:12.955928 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0595ee2f9d4442cca8e712debfae068e" } new_table_name: "default.Table_8115dcee1787457aab5b9023e08afe00" modify_external_catalogs: true
I20250110 02:51:12.956316 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_215c0fc0ef884696ad73db0829ca7e67" } new_table_name: "default.Table_123de8ace5ed47f7850040c5de4fcbc7" modify_external_catalogs: true
I20250110 02:51:12.956735 890 tablet_service.cc:1467] Processing CreateTablet for tablet 302af84fd8374ffa96a7503b54849536 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.956766 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0167bf0203804e32a12575dadcebdd98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:12.957651 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0167bf0203804e32a12575dadcebdd98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:12.958609 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_99c8d182fc924995a8f6e78bd31f07a6" } modify_external_catalogs: true
I20250110 02:51:12.959360 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_43eef7c99fbb4bffb1dc9eb5129cebae" } new_table_name: "default.Table_bf5730ffdb414255925a1360ec7d7626" modify_external_catalogs: true
I20250110 02:51:12.959610 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.960510 1427 raft_consensus.cc:491] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.960659 1427 raft_consensus.cc:513] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:12.961205 1427 leader_election.cc:290] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:12.961434 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_99c8d182fc924995a8f6e78bd31f07a6" } modify_external_catalogs: true
I20250110 02:51:12.961545 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 0127de45264f4660b843734a530eb310
I20250110 02:51:12.964052 526 catalog_manager.cc:6596] ReplaceTablet: tablet 0127de45264f4660b843734a530eb310 deleted and replaced by tablet 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:12.966946 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.967255 1184 tablet_bootstrap.cc:654] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.968741 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:12.969611 709 leader_election.cc:343] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:12.970304 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46a4745356e3423f9d4892d242da1c48 consensus state has no leader
I20250110 02:51:12.970329 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:12.942708 1034 tablet_service.cc:1467] Processing CreateTablet for tablet fb2272dfd0b14113afa3e13b38de6f8c (DEFAULT_TABLE table=default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 [id=9643d639878f4122bf42aeca200f9c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.971733 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:12.971808 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:12.971772 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:12.977361 711 leader_election.cc:343] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:12.977468 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95b14d7592024a34baf7756d92e546d9 consensus state has no leader
I20250110 02:51:12.977507 711 leader_election.cc:304] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:12.977802 1427 raft_consensus.cc:2743] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.978003 1367 raft_consensus.cc:491] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:12.978169 1367 raft_consensus.cc:513] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:12.978732 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 361c332d66984cf7ab84d74e913a5bde consensus state has no leader
I20250110 02:51:12.978925 1367 leader_election.cc:290] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:12.979875 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ab917bde534e3fbdf5dbae4af9304e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:12.980775 853 leader_election.cc:343] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 10ab917bde534e3fbdf5dbae4af9304e
I20250110 02:51:12.981335 1061 raft_consensus.cc:2388] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:12.981720 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 302af84fd8374ffa96a7503b54849536 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:12.982559 853 leader_election.cc:304] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:12.982964 1367 raft_consensus.cc:3054] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:12.983156 1184 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:12.983322 1184 ts_tablet_manager.cc:1397] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20250110 02:51:12.984169 1184 raft_consensus.cc:357] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.984421 1184 raft_consensus.cc:383] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:12.984510 1184 raft_consensus.cc:738] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:12.984959 1184 consensus_queue.cc:260] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:12.985769 1367 raft_consensus.cc:2743] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:12.986220 1184 ts_tablet_manager.cc:1428] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:12.986523 1184 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:12.988793 1184 tablet_bootstrap.cc:654] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:12.982221 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:12.994305 1386 raft_consensus.cc:2798] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:12.999264 743 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.000068 1184 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.000240 1184 ts_tablet_manager.cc:1397] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20250110 02:51:13.000275 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.001077 1178 tablet_replica.cc:331] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:13.001209 1184 raft_consensus.cc:357] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.001487 1184 raft_consensus.cc:383] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.001579 1184 raft_consensus.cc:738] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.002015 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.002360 1036 tablet_service.cc:1467] Processing CreateTablet for tablet abbd43638a2c47dcba4f6f90e8676f42 (DEFAULT_TABLE table=default.Table_50adace768fa44c698737e7843f5ac75 [id=edc4fd777f664491be5d464c39378878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.002338 1184 consensus_queue.cc:260] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.003079 1184 ts_tablet_manager.cc:1428] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:13.003437 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.003939 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.003942 745 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.006448 1184 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.009352 1184 tablet_bootstrap.cc:654] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.010221 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 130) took 2052 ms. Trace:
I20250110 02:51:13.010648 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971f97a3be9b49958b1218a275e126d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.011387 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3510e86314144a2ca012aacf3d78eacc (DEFAULT_TABLE table=default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.011695 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbd43638a2c47dcba4f6f90e8676f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.012964 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3510e86314144a2ca012aacf3d78eacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.015408 1180 rpcz_store.cc:276] 0110 02:51:10.868282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.868449 (+ 167us) service_pool.cc:225] Handling call
0110 02:51:12.903061 (+2034612us) ts_tablet_manager.cc:1145] Deleting tablet 423c820b81b845719d59a376c18e175a
0110 02:51:12.903070 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:12.920508 (+ 17438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2033900}
I20250110 02:51:13.015785 1178 raft_consensus.cc:2238] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.016049 1178 raft_consensus.cc:2267] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.016620 1178 ts_tablet_manager.cc:1905] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.019228 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4d2bb0b992474d8e5d21abaca5411c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.022033 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2272dfd0b14113afa3e13b38de6f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.025816 1386 raft_consensus.cc:695] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:13.026132 1386 consensus_queue.cc:237] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.027379 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e758fcd046f4196a3735d4b731544e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.029454 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302af84fd8374ffa96a7503b54849536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.032207 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8df25635dd7e4d58818dc14fc678e221" } modify_external_catalogs: true
I20250110 02:51:13.032478 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8df25635dd7e4d58818dc14fc678e221" } modify_external_catalogs: true
I20250110 02:51:13.038453 1180 tablet_replica.cc:331] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:13.038746 1180 raft_consensus.cc:2238] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.039178 1180 pending_rounds.cc:62] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:13.039827 1180 pending_rounds.cc:69] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611719745671168 op_type: NO_OP noop_request { }
I20250110 02:51:13.040396 1180 raft_consensus.cc:2883] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:13.040567 1180 raft_consensus.cc:2267] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:13.040926 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 423c820b81b845719d59a376c18e175a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 423c820b81b845719d59a376c18e175a already in progress: creating tablet
I20250110 02:51:13.043294 523 master_service.cc:946] ReplaceTablet: received request to replace tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.045378 1184 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.045531 1184 ts_tablet_manager.cc:1397] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20250110 02:51:13.046418 1184 raft_consensus.cc:357] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.046685 1184 raft_consensus.cc:383] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.046778 1184 raft_consensus.cc:738] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.046998 1184 consensus_queue.cc:260] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.047549 1184 ts_tablet_manager.cc:1428] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:13.047874 1184 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.051215 1184 tablet_bootstrap.cc:654] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.053329 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 (DEFAULT_TABLE table=default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.053705 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9289de0630804f9aa60fdaf5f3f15f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.054088 748 tablet_service.cc:1467] Processing CreateTablet for tablet a7397e41f7374e9fb1621dbcf30af703 (DEFAULT_TABLE table=default.Table_b0b62b5867624cd1b77150b7b3d2090d [id=a91e19e65fd74454b4b4d339a5cbe806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.054107 894 tablet_service.cc:1467] Processing CreateTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.054461 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7397e41f7374e9fb1621dbcf30af703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.055200 744 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9d524fda0d4848b333df9e20a9f68b (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.055958 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a9d524fda0d4848b333df9e20a9f68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.057621 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.058152 756 tablet_service.cc:1467] Processing CreateTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.058488 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f5f457d9e4028bfd0b707604c42f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.059278 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_28eb81570b2a4c609ee26c91c6aaf95e" } new_table_name: "default.Table_2b2430678ec9494e8179181b89afcd33" modify_external_catalogs: true
I20250110 02:51:13.064070 1179 ts_tablet_manager.cc:1905] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.066067 740 tablet_service.cc:1467] Processing CreateTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.067855 1367 raft_consensus.cc:491] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.068015 1367 raft_consensus.cc:513] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.066648 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e804fe33c8ae489bbc4d3765c64b59bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.069011 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_cebc63fd7b144a688a2c94924bbf5b08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.069404 1367 leader_election.cc:290] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:13.069527 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cebc63fd7b144a688a2c94924bbf5b08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.071856 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.072314 1179 ts_tablet_manager.cc:1918] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:13.072500 1179 log.cc:1198] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/02d3f60c819f4824bc99d755e7442976
I20250110 02:51:13.072602 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb4e06e40c43438882e1eb622d788e (DEFAULT_TABLE table=default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e [id=387a7fedac2a4e4998db6500fc8c9334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.072916 1179 ts_tablet_manager.cc:1939] T 02d3f60c819f4824bc99d755e7442976 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.073431 885 tablet_service.cc:1467] Processing CreateTablet for tablet 24c4042c38de4969889f4821176fc7bb (DEFAULT_TABLE table=default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.074139 879 tablet_service.cc:1467] Processing CreateTablet for tablet a81c031451104d5a93a1672b4b1918c5 (DEFAULT_TABLE table=default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.074786 1179 tablet_replica.cc:331] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.074963 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:13.075174 776 raft_consensus.cc:2463] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:13.075363 1179 raft_consensus.cc:2238] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.076046 1179 raft_consensus.cc:2267] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.076129 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b7d4e47c779b4a43a9e1719d181bf639" } new_table_name: "default.Table_7c5e74f9bb4e47a68454f6b42cf6ab2d" modify_external_catalogs: true
I20250110 02:51:13.076175 853 leader_election.cc:304] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:13.076480 1367 raft_consensus.cc:2798] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:13.076656 1367 raft_consensus.cc:491] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:13.076805 1367 raft_consensus.cc:3054] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.077939 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e (DEFAULT_TABLE table=default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.079241 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4d3270e3a04cc99f90943b9573ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.079785 1184 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.079968 1184 ts_tablet_manager.cc:1397] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.005s
I20250110 02:51:13.080885 1184 raft_consensus.cc:357] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.081139 1184 raft_consensus.cc:383] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.081224 1184 raft_consensus.cc:738] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.081492 1184 consensus_queue.cc:260] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.081792 880 tablet_service.cc:1467] Processing CreateTablet for tablet a7397e41f7374e9fb1621dbcf30af703 (DEFAULT_TABLE table=default.Table_b0b62b5867624cd1b77150b7b3d2090d [id=a91e19e65fd74454b4b4d339a5cbe806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.082129 1184 ts_tablet_manager.cc:1428] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:13.081971 1367 raft_consensus.cc:513] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.082512 1184 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.084478 1184 tablet_bootstrap.cc:654] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.085019 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a792a53d593141ddb875604e99d221d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.085875 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a792a53d593141ddb875604e99d221d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.086588 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f5c7f32764f54b818389fa6f51276bea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.087204 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5c7f32764f54b818389fa6f51276bea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.089838 1427 raft_consensus.cc:491] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.090008 1427 raft_consensus.cc:513] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.090770 1427 leader_election.cc:290] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.092016 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:13.092857 709 leader_election.cc:343] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.094249 1367 leader_election.cc:290] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.094317 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:13.094604 776 raft_consensus.cc:3054] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.095212 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.095582 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e25f9643ac0476a90c93fb9aeb921a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.096223 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb4e06e40c43438882e1eb622d788e (DEFAULT_TABLE table=default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e [id=387a7fedac2a4e4998db6500fc8c9334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.096637 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edb4e06e40c43438882e1eb622d788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.098634 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:13.098655 1179 ts_tablet_manager.cc:1905] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.099951 776 raft_consensus.cc:2463] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:13.100111 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:13.100355 853 leader_election.cc:304] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:13.100514 523 catalog_manager.cc:6596] ReplaceTablet: tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 deleted and replaced by tablet 815767a4c7624852bfedab9fa6cb5f44
I20250110 02:51:13.101542 756 tablet_service.cc:1467] Processing CreateTablet for tablet a81c031451104d5a93a1672b4b1918c5 (DEFAULT_TABLE table=default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.101876 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81c031451104d5a93a1672b4b1918c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.105329 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3510e86314144a2ca012aacf3d78eacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.105572 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81c031451104d5a93a1672b4b1918c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.105770 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f5f457d9e4028bfd0b707604c42f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.108928 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e25f9643ac0476a90c93fb9aeb921a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.110481 1184 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.110556 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4042c38de4969889f4821176fc7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.110628 1184 ts_tablet_manager.cc:1397] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20250110 02:51:13.111557 1184 raft_consensus.cc:357] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.111789 1184 raft_consensus.cc:383] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.111877 1184 raft_consensus.cc:738] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.112098 1184 consensus_queue.cc:260] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.112754 1184 ts_tablet_manager.cc:1428] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:13.113026 1184 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.114177 1367 raft_consensus.cc:2798] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:13.114929 1184 tablet_bootstrap.cc:654] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.115986 1188 tablet_bootstrap.cc:492] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:13.116132 1188 ts_tablet_manager.cc:1397] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.318s user 0.013s sys 0.037s
I20250110 02:51:13.116230 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a4b99ef64b874b609f835a3e58454edd" } modify_external_catalogs: true
I20250110 02:51:13.116425 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a4b99ef64b874b609f835a3e58454edd" } modify_external_catalogs: true
I20250110 02:51:13.117110 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_83ab4512090d4fd0b699700953b61a82" } modify_external_catalogs: true
I20250110 02:51:13.117256 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_83ab4512090d4fd0b699700953b61a82" } modify_external_catalogs: true
I20250110 02:51:13.117825 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edb4e06e40c43438882e1eb622d788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.117853 1188 raft_consensus.cc:357] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.118129 1188 raft_consensus.cc:383] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.118271 1188 raft_consensus.cc:738] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.119232 1188 consensus_queue.cc:260] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.120560 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 02d3f60c819f4824bc99d755e7442976 (table default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]) successfully deleted
W20250110 02:51:13.120894 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3fa76357abae4a95b8fb3af47288e43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa76357abae4a95b8fb3af47288e43a already in progress: creating tablet
W20250110 02:51:13.121109 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d9018955f3884824beb2e7336987fc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9018955f3884824beb2e7336987fc1c already in progress: creating tablet
I20250110 02:51:13.122123 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.122561 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.123266 1184 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.123420 1184 ts_tablet_manager.cc:1397] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20250110 02:51:13.124114 1184 raft_consensus.cc:357] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.124310 1184 raft_consensus.cc:383] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.124390 1184 raft_consensus.cc:738] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.124601 1184 consensus_queue.cc:260] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.126828 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7377066199724b26af78c183f9811324" } modify_external_catalogs: true
I20250110 02:51:13.127053 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7377066199724b26af78c183f9811324" } modify_external_catalogs: true
I20250110 02:51:13.127149 1184 ts_tablet_manager.cc:1428] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:13.127405 1184 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.127696 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c7fa3a12842444fa9aa1758777787789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.128188 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7fa3a12842444fa9aa1758777787789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.128767 1184 tablet_bootstrap.cc:654] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.129288 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_86a6941cdab84e57989003542ba3eb76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.129808 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86a6941cdab84e57989003542ba3eb76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.130179 749 tablet_service.cc:1467] Processing CreateTablet for tablet 302af84fd8374ffa96a7503b54849536 (DEFAULT_TABLE table=default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.130513 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302af84fd8374ffa96a7503b54849536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.137121 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302af84fd8374ffa96a7503b54849536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.137778 1188 ts_tablet_manager.cc:1428] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250110 02:51:13.138167 1188 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:13.141512 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971f97a3be9b49958b1218a275e126d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.143414 1188 tablet_bootstrap.cc:654] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:13.146404 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1390f867a3a8436092d1920daff4c5b0 consensus state has no leader
I20250110 02:51:13.146539 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2272dfd0b14113afa3e13b38de6f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.146715 517 master_service.cc:946] ReplaceTablet: received request to replace tablet b375a932156d4abb908dbf327689b0f9 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.147835 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c83271627bc9404393dd64f25709c4c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.148465 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c83271627bc9404393dd64f25709c4c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.149073 1179 ts_tablet_manager.cc:1918] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.151599 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7b5def1fa803454baf15dfee9b48fe75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.152019 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e4d2bb0b992474d8e5d21abaca5411c consensus state has no leader
W20250110 02:51:13.152061 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b5def1fa803454baf15dfee9b48fe75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.154006 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_71d270ae71ca41e384eb525387ffd768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.154667 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_051f5b2057e84287934aac189c8ea3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.155272 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_051f5b2057e84287934aac189c8ea3da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.160265 881 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9d524fda0d4848b333df9e20a9f68b (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.160761 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a9d524fda0d4848b333df9e20a9f68b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.163173 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71d270ae71ca41e384eb525387ffd768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.166325 895 tablet_service.cc:1467] Processing CreateTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.166702 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e804fe33c8ae489bbc4d3765c64b59bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.137084 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.172680 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_033d0d58d92e4a23ae7b1927e5db6163" } new_table_name: "default.Table_380ce069abd543e38d5fe587fcf166cb" modify_external_catalogs: true
I20250110 02:51:13.172737 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e7d7ae7a9eea4238802be676c6f59905" } modify_external_catalogs: true
I20250110 02:51:13.172905 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e7d7ae7a9eea4238802be676c6f59905" } modify_external_catalogs: true
I20250110 02:51:13.182039 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:13.182263 1039 tablet_service.cc:1467] Processing CreateTablet for tablet a7397e41f7374e9fb1621dbcf30af703 (DEFAULT_TABLE table=default.Table_b0b62b5867624cd1b77150b7b3d2090d [id=a91e19e65fd74454b4b4d339a5cbe806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.186486 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.187716 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 5a9d524fda0d4848b333df9e20a9f68b (DEFAULT_TABLE table=default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.188051 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.191148 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7397e41f7374e9fb1621dbcf30af703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.191411 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6bf83ecf29b84c6babc878760e92fce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.191879 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bf83ecf29b84c6babc878760e92fce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.195176 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 (DEFAULT_TABLE table=default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.195537 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7397e41f7374e9fb1621dbcf30af703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.196264 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 (DEFAULT_TABLE table=default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.197003 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cccf34a241bd4aaf9296a023cdd9a853" } new_table_name: "default.Table_ee284925f5344933a090d3f345f45b53" modify_external_catalogs: true
I20250110 02:51:13.197365 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 27a7885a80414f2193400672aa16a911
I20250110 02:51:13.198220 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.198218 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7c5e74f9bb4e47a68454f6b42cf6ab2d" } new_table_name: "default.Table_f10221708fa5417d8300ada3f8eb0817" modify_external_catalogs: true
I20250110 02:51:13.199625 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b375a932156d4abb908dbf327689b0f9
I20250110 02:51:13.200134 517 catalog_manager.cc:6596] ReplaceTablet: tablet b375a932156d4abb908dbf327689b0f9 deleted and replaced by tablet 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:13.200265 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a9d524fda0d4848b333df9e20a9f68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.200487 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 514704cd935b44e8b760221224332509
I20250110 02:51:13.200985 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.201170 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:13.201238 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9289de0630804f9aa60fdaf5f3f15f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.201169 891 tablet_service.cc:1467] Processing CreateTablet for tablet 4d970e2fe72b4727968ca605b3e772cc (DEFAULT_TABLE table=default.Table_0167bf0203804e32a12575dadcebdd98 [id=c584ebe680bb449e92795453c9b6ab74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.201787 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e (DEFAULT_TABLE table=default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.202610 885 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9ead3175140028e584e1ccb4874de (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.205957 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.206743 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e804fe33c8ae489bbc4d3765c64b59bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.208180 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c09235548b5d4e10a01cbe05d1c94841 consensus state has no leader
I20250110 02:51:13.209849 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.210105 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9289de0630804f9aa60fdaf5f3f15f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.210809 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 514704cd935b44e8b760221224332509 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.211628 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1edb4e06e40c43438882e1eb622d788e (DEFAULT_TABLE table=default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e [id=387a7fedac2a4e4998db6500fc8c9334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.212854 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.217226 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet a81c031451104d5a93a1672b4b1918c5
I20250110 02:51:13.171895 744 tablet_service.cc:1467] Processing CreateTablet for tablet 24c4042c38de4969889f4821176fc7bb (DEFAULT_TABLE table=default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.218670 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4042c38de4969889f4821176fc7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.220149 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.220906 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 514704cd935b44e8b760221224332509 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.221722 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.178419 746 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.179060 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4d970e2fe72b4727968ca605b3e772cc (DEFAULT_TABLE table=default.Table_0167bf0203804e32a12575dadcebdd98 [id=c584ebe680bb449e92795453c9b6ab74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.222724 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d970e2fe72b4727968ca605b3e772cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.223261 746 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.225737 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.179490 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9ead3175140028e584e1ccb4874de (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.179623 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.226194 1179 log.cc:1198] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/aa40d20288644b2a8d36ede967d16e79
I20250110 02:51:13.193557 745 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.209525 1314 raft_consensus.cc:491] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.227317 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb9ead3175140028e584e1ccb4874de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.227684 1314 raft_consensus.cc:513] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.228973 1314 leader_election.cc:290] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.229135 1314 raft_consensus.cc:491] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.229259 1314 raft_consensus.cc:513] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.230195 1314 leader_election.cc:290] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.189946 1184 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.231575 1184 ts_tablet_manager.cc:1397] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.023s
I20250110 02:51:13.232079 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9ead3175140028e584e1ccb4874de (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.232355 1184 raft_consensus.cc:357] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.232509 1184 raft_consensus.cc:383] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.232605 1184 raft_consensus.cc:738] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.232800 1184 consensus_queue.cc:260] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.232976 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.233178 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet 514704cd935b44e8b760221224332509 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.233309 1184 ts_tablet_manager.cc:1428] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:13.233321 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.233453 1032 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.233578 1184 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.233747 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4d970e2fe72b4727968ca605b3e772cc (DEFAULT_TABLE table=default.Table_0167bf0203804e32a12575dadcebdd98 [id=c584ebe680bb449e92795453c9b6ab74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.233984 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.234529 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:13.234840 711 leader_election.cc:343] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.234967 711 leader_election.cc:304] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.235098 1184 tablet_bootstrap.cc:654] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.235376 1427 raft_consensus.cc:2743] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.238997 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1602ade49b7442d6b8e6f4b85d2e80a1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.239475 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.241636 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edb4e06e40c43438882e1eb622d788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.242059 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d970e2fe72b4727968ca605b3e772cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.242820 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb9ead3175140028e584e1ccb4874de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.244998 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e25f9643ac0476a90c93fb9aeb921a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.246541 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 676f5f457d9e4028bfd0b707604c42f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.248224 1181 tablet_bootstrap.cc:492] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:13.269053 1179 ts_tablet_manager.cc:1939] T aa40d20288644b2a8d36ede967d16e79 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.269935 1179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55720 (request call id 210) took 1065 ms. Trace:
I20250110 02:51:13.271090 1179 rpcz_store.cc:276] 0110 02:51:12.204446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.204500 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:13.074675 (+870175us) ts_tablet_manager.cc:1145] Deleting tablet aa40d20288644b2a8d36ede967d16e79
0110 02:51:13.074684 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.146198 (+ 71514us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.148972 (+ 2774us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.269894 (+120922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":869909}
I20250110 02:51:13.271601 1179 tablet_replica.cc:331] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:13.272197 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2ef72e05f8ed4d46b759c7a04497653b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ef72e05f8ed4d46b759c7a04497653b already in progress: creating tablet
I20250110 02:51:13.272615 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 24c4042c38de4969889f4821176fc7bb (DEFAULT_TABLE table=default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.275760 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4d3270e3a04cc99f90943b9573ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.278272 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1390f867a3a8436092d1920daff4c5b0 consensus state has no leader
W20250110 02:51:13.278709 711 leader_election.cc:343] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.278944 711 leader_election.cc:343] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.280210 1179 raft_consensus.cc:2238] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.280517 756 tablet_service.cc:1467] Processing CreateTablet for tablet b382d634db9642df8e00c5ccaa5787a9 (DEFAULT_TABLE table=default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.281301 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b382d634db9642df8e00c5ccaa5787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.282548 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (DEFAULT_TABLE table=default.Table_cebc63fd7b144a688a2c94924bbf5b08 [id=28ef0a56c1054854a22896fb038a2aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.282672 524 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_c12448a050c84ad3aa468fe82cc4337b [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 2 got 0
I20250110 02:51:13.282846 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.284178 1034 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (DEFAULT_TABLE table=default.Table_cebc63fd7b144a688a2c94924bbf5b08 [id=28ef0a56c1054854a22896fb038a2aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.284292 1179 raft_consensus.cc:2267] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.284873 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (DEFAULT_TABLE table=default.Table_f5c7f32764f54b818389fa6f51276bea [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.199047 1180 ts_tablet_manager.cc:1905] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.286441 1179 ts_tablet_manager.cc:1905] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:13.287906 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e79175853474cbea951f5a5686bf630 consensus state has no leader
I20250110 02:51:13.288908 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 815767a4c7624852bfedab9fa6cb5f44 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.289158 743 tablet_service.cc:1467] Processing CreateTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (DEFAULT_TABLE table=default.Table_f5c7f32764f54b818389fa6f51276bea [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.289611 745 tablet_service.cc:1467] Processing CreateTablet for tablet 00ebf8debb9d4394839d1c1cb97820a5 (DEFAULT_TABLE table=default.Table_a792a53d593141ddb875604e99d221d1 [id=f6b39e5377964d51adc9632b8199e4f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.290328 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.290268 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e (DEFAULT_TABLE table=default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.290539 1184 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.291325 748 tablet_service.cc:1467] Processing CreateTablet for tablet 815767a4c7624852bfedab9fa6cb5f44 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.292142 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.292644 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d970e2fe72b4727968ca605b3e772cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.292608 1036 tablet_service.cc:1467] Processing CreateTablet for tablet a81c031451104d5a93a1672b4b1918c5 (DEFAULT_TABLE table=default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:13.294678 853 leader_election.cc:343] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.296032 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.296172 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4042c38de4969889f4821176fc7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.296648 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.297405 1181 ts_tablet_manager.cc:1397] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.585s user 0.006s sys 0.016s
I20250110 02:51:13.298077 1030 tablet_service.cc:1467] Processing CreateTablet for tablet b382d634db9642df8e00c5ccaa5787a9 (DEFAULT_TABLE table=default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.299188 1184 ts_tablet_manager.cc:1397] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.038s
I20250110 02:51:13.299371 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.299868 879 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.302369 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7b5def1fa803454baf15dfee9b48fe75" } modify_external_catalogs: true
I20250110 02:51:13.302611 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7b5def1fa803454baf15dfee9b48fe75" } modify_external_catalogs: true
I20250110 02:51:13.304298 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 00ebf8debb9d4394839d1c1cb97820a5 (DEFAULT_TABLE table=default.Table_a792a53d593141ddb875604e99d221d1 [id=f6b39e5377964d51adc9632b8199e4f5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:13.307276 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 372755us
I20250110 02:51:13.308637 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815767a4c7624852bfedab9fa6cb5f44. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.310853 853 leader_election.cc:343] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.311754 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.313992 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:13.315375 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb9ead3175140028e584e1ccb4874de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.316077 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebf8debb9d4394839d1c1cb97820a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.317131 740 tablet_service.cc:1467] Processing CreateTablet for tablet 75ca5369caf3415ea508352362409656 (DEFAULT_TABLE table=default.Table_6bf83ecf29b84c6babc878760e92fce1 [id=f84d6095952546dd94aedb2f33aa490e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.317939 744 tablet_service.cc:1467] Processing CreateTablet for tablet 452223af5f6a4836a0f582d394a9453e (DEFAULT_TABLE table=default.Table_71d270ae71ca41e384eb525387ffd768 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.318621 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1602ade49b7442d6b8e6f4b85d2e80a1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.319804 746 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.319765 1184 raft_consensus.cc:357] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.320292 1184 raft_consensus.cc:383] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.321156 1181 raft_consensus.cc:357] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.321722 1040 tablet_service.cc:1467] Processing CreateTablet for tablet 452223af5f6a4836a0f582d394a9453e (DEFAULT_TABLE table=default.Table_71d270ae71ca41e384eb525387ffd768 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.322415 1184 raft_consensus.cc:738] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.322901 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2c7a313ba94e5ea85e45276658ba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.323390 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.323554 752 tablet_service.cc:1467] Processing CreateTablet for tablet d424b4da66184bd19eccefad80621516 (DEFAULT_TABLE table=default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.323601 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 75ca5369caf3415ea508352362409656 (DEFAULT_TABLE table=default.Table_6bf83ecf29b84c6babc878760e92fce1 [id=f84d6095952546dd94aedb2f33aa490e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.323927 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.324028 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815767a4c7624852bfedab9fa6cb5f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.325138 754 tablet_service.cc:1467] Processing CreateTablet for tablet c1ade70d7fe34776bef3c5853f8dde74 (DEFAULT_TABLE table=default.Table_c7fa3a12842444fa9aa1758777787789 [id=88412101305b4d0a8dea71c08ff391aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.325472 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4af23720048fe98b3231ba402d45c (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.325767 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.326153 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4d3270e3a04cc99f90943b9573ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.326539 753 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (DEFAULT_TABLE table=default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.326071 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (DEFAULT_TABLE table=default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.327745 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d424b4da66184bd19eccefad80621516 (DEFAULT_TABLE table=default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.329020 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4af23720048fe98b3231ba402d45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.329344 755 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4af23720048fe98b3231ba402d45c (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.329510 1032 tablet_service.cc:1467] Processing CreateTablet for tablet c1ade70d7fe34776bef3c5853f8dde74 (DEFAULT_TABLE table=default.Table_c7fa3a12842444fa9aa1758777787789 [id=88412101305b4d0a8dea71c08ff391aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.328325 1184 consensus_queue.cc:260] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.330245 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2c7a313ba94e5ea85e45276658ba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.332198 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452223af5f6a4836a0f582d394a9453e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.333405 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_c12448a050c84ad3aa468fe82cc4337b [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 2 got 0
I20250110 02:51:13.334648 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424b4da66184bd19eccefad80621516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.334689 1181 raft_consensus.cc:383] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.334810 1181 raft_consensus.cc:738] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.335703 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424b4da66184bd19eccefad80621516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.336748 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de9cf980d9c4117b4ec3ca937cfcf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.337617 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4af23720048fe98b3231ba402d45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.338434 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ade70d7fe34776bef3c5853f8dde74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.339867 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ca5369caf3415ea508352362409656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.347599 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_380ce069abd543e38d5fe587fcf166cb [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 1 got 0
I20250110 02:51:13.348425 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_65e2869747e148ca99fe45b205855ef9" } new_table_name: "default.Table_90e43feb19a343f6b3c9a20876ec8c23" modify_external_catalogs: true
I20250110 02:51:13.350147 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3c02731f843a4cc3b13b54dee5b9e7f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.350677 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.351675 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452223af5f6a4836a0f582d394a9453e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.353013 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b382d634db9642df8e00c5ccaa5787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.355062 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_627a9df6d7b84326bf7e90f5423ff267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.355132 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c12448a050c84ad3aa468fe82cc4337b" } new_table_name: "default.Table_99ac393aee99471a8a4a750750f72cc8" modify_external_catalogs: true
W20250110 02:51:13.355602 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_627a9df6d7b84326bf7e90f5423ff267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.356401 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1fd67f09eeae41fdb559eabdb1d2fc82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.356582 1181 consensus_queue.cc:260] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:13.357576 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.360137 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81c031451104d5a93a1672b4b1918c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.360113 1427 raft_consensus.cc:491] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.360257 1427 raft_consensus.cc:513] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.360697 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ade70d7fe34776bef3c5853f8dde74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.360952 1427 leader_election.cc:290] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.368933 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e8d197a297e647b5ad3b516ff8054704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.369374 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d197a297e647b5ad3b516ff8054704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:13.369998 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da8f247c862e4cb68452e230857ee6b0 consensus state has no leader
I20250110 02:51:13.372768 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.373239 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebf8debb9d4394839d1c1cb97820a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.374997 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de9cf980d9c4117b4ec3ca937cfcf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.375438 524 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:13.375578 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ca5369caf3415ea508352362409656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.377076 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_87bb664f5b24476b9dd295e4fb7b7f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.377365 711 leader_election.cc:343] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.377573 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bb664f5b24476b9dd295e4fb7b7f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.379113 898 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.380836 890 tablet_service.cc:1467] Processing CreateTablet for tablet b382d634db9642df8e00c5ccaa5787a9 (DEFAULT_TABLE table=default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.381327 897 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.381399 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3b6669107e9943cc99dd6d48cd80c5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.381642 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2c7a313ba94e5ea85e45276658ba04. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.381927 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b6669107e9943cc99dd6d48cd80c5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.384461 894 tablet_service.cc:1467] Processing CreateTablet for tablet 00ebf8debb9d4394839d1c1cb97820a5 (DEFAULT_TABLE table=default.Table_a792a53d593141ddb875604e99d221d1 [id=f6b39e5377964d51adc9632b8199e4f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.385097 886 tablet_service.cc:1467] Processing CreateTablet for tablet 815767a4c7624852bfedab9fa6cb5f44 (DEFAULT_TABLE table=default.Table_3832746ca2454627a4b2495825f5223c [id=7b6faf943b984803b3086d2128a42462]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.385457 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.385911 1181 ts_tablet_manager.cc:1428] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.088s user 0.001s sys 0.002s
W20250110 02:51:13.387667 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f229f546a8d4a09b93f8b46c893d14a consensus state has no leader
I20250110 02:51:13.385627 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b382d634db9642df8e00c5ccaa5787a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.385766 881 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (DEFAULT_TABLE table=default.Table_cebc63fd7b144a688a2c94924bbf5b08 [id=28ef0a56c1054854a22896fb038a2aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.386714 895 tablet_service.cc:1467] Processing CreateTablet for tablet 75ca5369caf3415ea508352362409656 (DEFAULT_TABLE table=default.Table_6bf83ecf29b84c6babc878760e92fce1 [id=f84d6095952546dd94aedb2f33aa490e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.386724 882 tablet_service.cc:1467] Processing CreateTablet for tablet d424b4da66184bd19eccefad80621516 (DEFAULT_TABLE table=default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.386731 892 tablet_service.cc:1467] Processing CreateTablet for tablet 452223af5f6a4836a0f582d394a9453e (DEFAULT_TABLE table=default.Table_71d270ae71ca41e384eb525387ffd768 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.387701 879 tablet_service.cc:1467] Processing CreateTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (DEFAULT_TABLE table=default.Table_f5c7f32764f54b818389fa6f51276bea [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.388341 888 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4af23720048fe98b3231ba402d45c (DEFAULT_TABLE table=default.Table_526de8fc3ad6495ea9ec7e55e2accd58 [id=5e514acaba2d498a8724032d6bbc65d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.393649 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebf8debb9d4394839d1c1cb97820a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.395632 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ad6ed08b6a0e47599bd592fbaf2c5090"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.396124 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.398840 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424b4da66184bd19eccefad80621516. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.400162 709 leader_election.cc:343] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.400360 709 leader_election.cc:304] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:13.400777 709 leader_election.cc:343] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.400889 709 leader_election.cc:304] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.401234 1314 raft_consensus.cc:2743] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.401407 1314 raft_consensus.cc:2743] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.401849 1184 ts_tablet_manager.cc:1428] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.037s user 0.002s sys 0.002s
I20250110 02:51:13.405187 1179 ts_tablet_manager.cc:1918] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.405546 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452223af5f6a4836a0f582d394a9453e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.407821 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a95ba565dbf940f79ed6f8b2eff1e74b" } modify_external_catalogs: true
I20250110 02:51:13.408027 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a95ba565dbf940f79ed6f8b2eff1e74b" } modify_external_catalogs: true
I20250110 02:51:13.408149 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d9338439b43445e9b19ed1f53c53d149" } modify_external_catalogs: true
I20250110 02:51:13.408280 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d9338439b43445e9b19ed1f53c53d149" } modify_external_catalogs: true
I20250110 02:51:13.409162 1179 log.cc:1198] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/423c820b81b845719d59a376c18e175a
I20250110 02:51:13.409801 1179 ts_tablet_manager.cc:1939] T 423c820b81b845719d59a376c18e175a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.410096 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7e7300d4fc9b49639885bbf4895cd07f" } new_table_name: "default.Table_bfc621ed30834a669b0ba54dcda3b490" modify_external_catalogs: true
I20250110 02:51:13.410682 1184 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.411140 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_48581fba686f44bea7a67b12a11e149f" } modify_external_catalogs: true
I20250110 02:51:13.411294 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_48581fba686f44bea7a67b12a11e149f" } modify_external_catalogs: true
I20250110 02:51:13.412937 1184 tablet_bootstrap.cc:654] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.413231 896 tablet_service.cc:1467] Processing CreateTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (DEFAULT_TABLE table=default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.416365 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4af23720048fe98b3231ba402d45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.417541 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3ceaee2d997344448ebb32110efcfcf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.418072 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ceaee2d997344448ebb32110efcfcf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.419013 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 423c820b81b845719d59a376c18e175a (table default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]) successfully deleted
W20250110 02:51:13.420548 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5e183122d13a4c38988bf2fe6f7a3694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e183122d13a4c38988bf2fe6f7a3694 already in progress: creating tablet
W20250110 02:51:13.421196 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7d4422eca5604d149e73c2a669d07af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d4422eca5604d149e73c2a669d07af0 already in progress: creating tablet
W20250110 02:51:13.421932 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fc9826ef554a5699e365d10d02e97a already in progress: creating tablet
I20250110 02:51:13.419314 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.423169 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:13.423283 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dad0d561a8ef498ba13bc813d4cd82fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dad0d561a8ef498ba13bc813d4cd82fa already in progress: creating tablet
W20250110 02:51:13.423995 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d20177229f7b4f11875e0007afb1dd00 already in progress: creating tablet
I20250110 02:51:13.424330 1179 tablet_replica.cc:331] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.424557 1179 raft_consensus.cc:2238] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.419656 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ecf7ad441f824cbe9e7384fb02a46754" } modify_external_catalogs: true
I20250110 02:51:13.424737 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ecf7ad441f824cbe9e7384fb02a46754" } modify_external_catalogs: true
I20250110 02:51:13.424789 1179 raft_consensus.cc:2267] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.425585 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:13.426321 709 leader_election.cc:343] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.426477 709 leader_election.cc:304] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.426919 1427 raft_consensus.cc:2743] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.428560 1479 raft_consensus.cc:491] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:13.430665 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa40c9c3f0ef447294b7c1c5f29ba06e consensus state has no leader
I20250110 02:51:13.430756 1179 ts_tablet_manager.cc:1905] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.431751 526 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_380ce069abd543e38d5fe587fcf166cb [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 1 got 0
I20250110 02:51:13.428720 1479 raft_consensus.cc:513] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:13.436861 511 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8e79175853474cbea951f5a5686bf630 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8e79175853474cbea951f5a5686bf630 consensus state has no leader
I20250110 02:51:13.437942 745 tablet_service.cc:1467] Processing CreateTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 (DEFAULT_TABLE table=default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.438282 1479 leader_election.cc:290] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.438457 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e05a7554a14805bdcbd1824a5bd9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.438696 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de9cf980d9c4117b4ec3ca937cfcf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.438982 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2412420dc54258a6921ead194c1587" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:13.439481 853 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.443353 853 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0c2412420dc54258a6921ead194c1587
I20250110 02:51:13.446069 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.446267 1184 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.446682 1184 ts_tablet_manager.cc:1397] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.005s
I20250110 02:51:13.447366 882 tablet_service.cc:1467] Processing CreateTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 (DEFAULT_TABLE table=default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.447470 1184 raft_consensus.cc:357] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.447672 1184 raft_consensus.cc:383] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.447788 1184 raft_consensus.cc:738] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.448010 1184 consensus_queue.cc:260] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.448613 853 leader_election.cc:304] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:13.449249 1479 raft_consensus.cc:2743] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.450465 1184 ts_tablet_manager.cc:1428] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:13.451234 1184 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.454902 1184 tablet_bootstrap.cc:654] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.455384 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.457440 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815767a4c7624852bfedab9fa6cb5f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.457743 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8c3ce95413df40369ee599176f91b892
I20250110 02:51:13.458484 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:13.459009 1179 ts_tablet_manager.cc:1918] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.459123 1179 log.cc:1198] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1602ade49b7442d6b8e6f4b85d2e80a1
I20250110 02:51:13.459177 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:13.460084 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.460906 745 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.461274 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.463048 898 tablet_service.cc:1467] Processing CreateTablet for tablet c1ade70d7fe34776bef3c5853f8dde74 (DEFAULT_TABLE table=default.Table_c7fa3a12842444fa9aa1758777787789 [id=88412101305b4d0a8dea71c08ff391aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.466944 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 (DEFAULT_TABLE table=default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.469717 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_bb82c27076c5492e9788b3e1b4a66c1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.470255 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb82c27076c5492e9788b3e1b4a66c1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.470633 1179 ts_tablet_manager.cc:1939] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.471477 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 302af84fd8374ffa96a7503b54849536
I20250110 02:51:13.472703 1179 tablet_replica.cc:331] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.472982 1179 raft_consensus.cc:2238] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.473196 1179 raft_consensus.cc:2267] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.473712 1179 ts_tablet_manager.cc:1905] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.474951 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ca5369caf3415ea508352362409656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.479367 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e05a7554a14805bdcbd1824a5bd9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.480160 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.480192 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.480471 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.482321 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.483088 1179 ts_tablet_manager.cc:1918] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.485073 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f5c7f32764f54b818389fa6f51276bea" } new_table_name: "default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2" modify_external_catalogs: true
I20250110 02:51:13.485636 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.489179 1179 log.cc:1198] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/0b74812cb81c44fcb8786cc5dae328e3
I20250110 02:51:13.489410 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a4b11eff25ef415cbdd62ed46e34ddcf" } new_table_name: "default.Table_c8632cb324c745eca03df9733d23a9a7" modify_external_catalogs: true
I20250110 02:51:13.489709 1179 ts_tablet_manager.cc:1939] T 0b74812cb81c44fcb8786cc5dae328e3 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.492765 1178 ts_tablet_manager.cc:1918] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:13.492893 1178 log.cc:1198] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/308f307f4200409aaa47df55f20472d0
I20250110 02:51:13.493294 1178 ts_tablet_manager.cc:1939] T 308f307f4200409aaa47df55f20472d0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:13.493559 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 167) took 1792 ms. Trace:
I20250110 02:51:13.493832 1178 rpcz_store.cc:276] 0110 02:51:11.700829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.700877 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:13.000853 (+1299976us) ts_tablet_manager.cc:1145] Deleting tablet 308f307f4200409aaa47df55f20472d0
0110 02:51:13.000864 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.277942 (+277078us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.489514 (+211572us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.493530 (+ 4016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255547}
I20250110 02:51:13.494331 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
I20250110 02:51:13.494634 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 174) took 1721 ms. Trace:
I20250110 02:51:13.494849 1178 rpcz_store.cc:276] 0110 02:51:11.773463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.773521 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:13.494359 (+1720838us) ts_tablet_manager.cc:1145] Deleting tablet 019c0d60da5548bb86c93ccd89bf94ff
0110 02:51:13.494401 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.494563 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693922}
W20250110 02:51:13.495429 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 019c0d60da5548bb86c93ccd89bf94ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019c0d60da5548bb86c93ccd89bf94ff already in progress: creating tablet
I20250110 02:51:13.495733 1179 tablet_replica.cc:331] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.495726 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 175) took 1694 ms. Trace:
I20250110 02:51:13.496138 1179 raft_consensus.cc:2238] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.495959 1178 rpcz_store.cc:276] 0110 02:51:11.801275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.802046 (+ 771us) service_pool.cc:225] Handling call
0110 02:51:13.495512 (+1693466us) ts_tablet_manager.cc:1145] Deleting tablet f14f06ba7d0f44418693e36f6bc6d888
0110 02:51:13.495520 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.495662 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692666}
W20250110 02:51:13.496526 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f14f06ba7d0f44418693e36f6bc6d888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14f06ba7d0f44418693e36f6bc6d888 already in progress: creating tablet
I20250110 02:51:13.496640 1179 raft_consensus.cc:2267] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.496825 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 181) took 1625 ms. Trace:
I20250110 02:51:13.497184 1179 ts_tablet_manager.cc:1905] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.497040 1178 rpcz_store.cc:276] 0110 02:51:11.871621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.882413 (+ 10792us) service_pool.cc:225] Handling call
0110 02:51:13.496549 (+1614136us) ts_tablet_manager.cc:1145] Deleting tablet 48cb20d47ace4fd49f9dc4e526cab702
0110 02:51:13.496594 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.496759 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613907}
W20250110 02:51:13.497589 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 48cb20d47ace4fd49f9dc4e526cab702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48cb20d47ace4fd49f9dc4e526cab702 already in progress: creating tablet
I20250110 02:51:13.499961 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.500922 1427 raft_consensus.cc:491] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.501071 1427 raft_consensus.cc:513] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:13.501242 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcc9a4a842914a6f878d01194cd9a5f7 already in progress: creating tablet
I20250110 02:51:13.501657 1427 leader_election.cc:290] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:13.501717 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 344184e1c98d422095cb254f5b2bc792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344184e1c98d422095cb254f5b2bc792 already in progress: creating tablet
I20250110 02:51:13.502197 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 0b74812cb81c44fcb8786cc5dae328e3 (table default.Table_2cb0f90ebf7248e1a9f43302db530a34 [id=c501975e25c34817b2c02a0e538c97a7]) successfully deleted
I20250110 02:51:13.476987 1180 ts_tablet_manager.cc:1918] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:13.504678 1180 log.cc:1198] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/05d5f9bda55942ff856252bbb50b3c5b
I20250110 02:51:13.506763 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.507983 1180 ts_tablet_manager.cc:1939] T 05d5f9bda55942ff856252bbb50b3c5b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:13.508656 709 leader_election.cc:343] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.509361 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e05a7554a14805bdcbd1824a5bd9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.511606 1367 raft_consensus.cc:695] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:13.515058 1427 raft_consensus.cc:491] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.515185 1427 raft_consensus.cc:513] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.515802 1427 leader_election.cc:290] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.516356 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ade70d7fe34776bef3c5853f8dde74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.516851 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1fd67f09eeae41fdb559eabdb1d2fc82" } modify_external_catalogs: true
I20250110 02:51:13.517066 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1fd67f09eeae41fdb559eabdb1d2fc82" } modify_external_catalogs: true
I20250110 02:51:13.468331 1181 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:13.492019 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.508234 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:13.520736 711 leader_election.cc:343] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.520913 711 leader_election.cc:304] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.521210 1427 raft_consensus.cc:2743] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.508884 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.509814 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.509997 1033 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.516983 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:13.522253 711 leader_election.cc:343] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.522480 1179 ts_tablet_manager.cc:1918] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.522958 1179 log.cc:1198] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/4e2c01af512f460daa98b19444f72172
I20250110 02:51:13.523052 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:13.526132 709 leader_election.cc:343] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.526244 709 leader_election.cc:304] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.526846 1427 raft_consensus.cc:2743] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.527333 1179 ts_tablet_manager.cc:1939] T 4e2c01af512f460daa98b19444f72172 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.533565 1179 tablet_replica.cc:331] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.533996 1179 raft_consensus.cc:2238] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.534413 1179 raft_consensus.cc:2267] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.535156 1179 ts_tablet_manager.cc:1905] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.540406 1386 raft_consensus.cc:491] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.541486 1179 ts_tablet_manager.cc:1918] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.541371 1386 raft_consensus.cc:513] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.541616 1179 log.cc:1198] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1dfc062b5fcb42c3bce6dfaed54b3869
I20250110 02:51:13.541961 1179 ts_tablet_manager.cc:1939] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.542452 1386 leader_election.cc:290] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.542801 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:13.543972 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 132) took 2650 ms. Trace:
I20250110 02:51:13.545656 1181 tablet_bootstrap.cc:654] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.546541 1367 consensus_queue.cc:237] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.546099 1180 rpcz_store.cc:276] 0110 02:51:10.869318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.869381 (+ 63us) service_pool.cc:225] Handling call
0110 02:51:13.015746 (+2146365us) ts_tablet_manager.cc:1145] Deleting tablet 05d5f9bda55942ff856252bbb50b3c5b
0110 02:51:13.015754 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.366613 (+350859us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.476863 (+110250us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.519460 (+ 42597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2146102}
I20250110 02:51:13.547847 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 131) took 2678 ms. Trace:
I20250110 02:51:13.547956 1180 rpcz_store.cc:276] 0110 02:51:10.869015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.869966 (+ 951us) service_pool.cc:225] Handling call
0110 02:51:13.547441 (+2677475us) ts_tablet_manager.cc:1145] Deleting tablet 2abd85a070f74e9bb5dc4026935edcb5
0110 02:51:13.547449 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.547744 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2677167}
I20250110 02:51:13.548638 1180 tablet_replica.cc:331] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:13.548861 1180 raft_consensus.cc:2238] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.549216 1180 raft_consensus.cc:2267] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.549520 776 raft_consensus.cc:2388] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8899dd1aeb384e4ea4523f5fd225c888 in current term 1: Already voted for candidate 85f9597077934f1a9814d284ed06d13d in this term.
I20250110 02:51:13.549994 1180 ts_tablet_manager.cc:1905] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.556303 1179 tablet_replica.cc:331] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.556538 1179 raft_consensus.cc:2238] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.561101 1179 raft_consensus.cc:2267] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.561620 1179 ts_tablet_manager.cc:1905] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:13.562707 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f4778347f054b86af697dbe3b287263 already in progress: creating tablet
W20250110 02:51:13.562947 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f5e88d1bbc304b4ebbc849a7d783052e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e88d1bbc304b4ebbc849a7d783052e already in progress: creating tablet
W20250110 02:51:13.563144 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b6d57f3e9a44d178300d3505d504854 already in progress: creating tablet
W20250110 02:51:13.563300 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d5ddbc6b02742609bc2ca84c863a9d0 already in progress: creating tablet
W20250110 02:51:13.563477 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f66603a0ee8c4082807c5a6aabe7e339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f66603a0ee8c4082807c5a6aabe7e339 already in progress: creating tablet
W20250110 02:51:13.564148 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 already in progress: creating tablet
I20250110 02:51:13.564353 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 4e2c01af512f460daa98b19444f72172 (table default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]) successfully deleted
I20250110 02:51:13.564517 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
W20250110 02:51:13.564718 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 32dd1e726f1a41d0894737b3ef3caf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32dd1e726f1a41d0894737b3ef3caf8f already in progress: creating tablet
I20250110 02:51:13.566147 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 05d5f9bda55942ff856252bbb50b3c5b (table default.Table_3e5d4c967ed04133a245f4185b257710 [id=bee875a9b8b2438db4aecd5f82a5044d]) successfully deleted
I20250110 02:51:13.569119 1184 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.569265 1184 ts_tablet_manager.cc:1397] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.118s user 0.017s sys 0.015s
I20250110 02:51:13.569962 1184 raft_consensus.cc:357] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.570207 1184 raft_consensus.cc:383] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.570281 1184 raft_consensus.cc:738] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.570560 1184 consensus_queue.cc:260] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.571630 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a037268bb40540c7b7495367b363d30b" } new_table_name: "default.Table_4c7ba976683d45708a4407a272ee2986" modify_external_catalogs: true
I20250110 02:51:13.572055 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 190) took 1615 ms. Trace:
I20250110 02:51:13.572819 1184 ts_tablet_manager.cc:1428] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:13.573119 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_dd0bd0decd37446f891e12d29ab4955b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.573577 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd0bd0decd37446f891e12d29ab4955b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.573635 1184 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.575346 1184 tablet_bootstrap.cc:654] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.582846 1179 ts_tablet_manager.cc:1918] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.582957 1179 log.cc:1198] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:13.583293 1179 ts_tablet_manager.cc:1939] T ad9f920cde954f31acebc6d0b6965e58 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.585167 1179 tablet_replica.cc:331] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.585351 1179 raft_consensus.cc:2238] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.585507 1179 raft_consensus.cc:2267] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.585695 1386 raft_consensus.cc:491] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.585883 1386 raft_consensus.cc:513] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.584213 1178 rpcz_store.cc:276] 0110 02:51:11.945770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.945814 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:13.498646 (+1552832us) ts_tablet_manager.cc:1145] Deleting tablet 64c77ec8c7724d5fb408a8bfa53811eb
0110 02:51:13.498659 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.561257 (+ 62598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1543759}
I20250110 02:51:13.586172 1188 tablet_bootstrap.cc:492] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:13.586397 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 64c77ec8c7724d5fb408a8bfa53811eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64c77ec8c7724d5fb408a8bfa53811eb already in progress: creating tablet
I20250110 02:51:13.586535 1386 leader_election.cc:290] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.586999 1188 ts_tablet_manager.cc:1397] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.448s user 0.009s sys 0.006s
I20250110 02:51:13.587564 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 194) took 1576 ms. Trace:
I20250110 02:51:13.587919 1178 rpcz_store.cc:276] 0110 02:51:12.009959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.010141 (+ 182us) service_pool.cc:225] Handling call
0110 02:51:13.586367 (+1576226us) ts_tablet_manager.cc:1145] Deleting tablet f9dbb61a51f14a3f96cc3c77981e8ce5
0110 02:51:13.586392 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.586505 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576041}
W20250110 02:51:13.588241 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2abd85a070f74e9bb5dc4026935edcb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2abd85a070f74e9bb5dc4026935edcb5 already in progress: creating tablet
I20250110 02:51:13.589360 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet ad9f920cde954f31acebc6d0b6965e58 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
W20250110 02:51:13.589643 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0127de45264f4660b843734a530eb310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0127de45264f4660b843734a530eb310 already in progress: creating tablet
I20250110 02:51:13.589337 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2bc2ba61f91d4e01ad04875d5e28c79e" } new_table_name: "default.Table_c9035b932c4f40428820b9bfa2d4ca61" modify_external_catalogs: true
W20250110 02:51:13.590516 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9dbb61a51f14a3f96cc3c77981e8ce5 already in progress: creating tablet
I20250110 02:51:13.590970 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c42eef47c4634ced8741d5d8bbb6b958"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.591459 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c42eef47c4634ced8741d5d8bbb6b958 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.592954 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3a64d77e95f7457490d8a6641411e1dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.593420 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a64d77e95f7457490d8a6641411e1dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:13.594000 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff9b10c419242cf85ef1b708bd5a862 already in progress: creating tablet
W20250110 02:51:13.594529 996 leader_election.cc:343] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: aa40d20288644b2a8d36ede967d16e79
W20250110 02:51:13.599911 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 consensus state has no leader
W20250110 02:51:13.600334 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e10e7654cec495bb532fb00e73c020d consensus state has no leader
I20250110 02:51:13.602146 1181 tablet_bootstrap.cc:492] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:13.602330 1181 ts_tablet_manager.cc:1397] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.134s user 0.005s sys 0.005s
I20250110 02:51:13.603209 1181 raft_consensus.cc:357] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.603453 1181 raft_consensus.cc:383] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.603588 1181 raft_consensus.cc:738] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.603842 1181 consensus_queue.cc:260] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.604532 1181 ts_tablet_manager.cc:1428] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250110 02:51:13.604554 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0d4d6e02b1d4334a9993d320217f1f8 consensus state has no leader
I20250110 02:51:13.605253 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]). Expected version 1 got 0
I20250110 02:51:13.605463 1181 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:13.606860 517 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:13.608283 1181 tablet_bootstrap.cc:654] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.614616 1180 ts_tablet_manager.cc:1918] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.614753 1180 log.cc:1198] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/66f6f4f221e6445db0bb569f372cfaa0
I20250110 02:51:13.615162 1180 ts_tablet_manager.cc:1939] T 66f6f4f221e6445db0bb569f372cfaa0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:13.615017 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0413ddbebc534e0da2f874ae0ecba335"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.615664 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0413ddbebc534e0da2f874ae0ecba335 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.615967 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 125) took 2769 ms. Trace:
I20250110 02:51:13.616093 1180 rpcz_store.cc:276] 0110 02:51:10.846814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.872189 (+ 25375us) service_pool.cc:225] Handling call
0110 02:51:13.548331 (+2676142us) ts_tablet_manager.cc:1145] Deleting tablet 66f6f4f221e6445db0bb569f372cfaa0
0110 02:51:13.548341 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.560414 (+ 12073us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.614451 (+ 54037us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.615923 (+ 1472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2675934}
I20250110 02:51:13.616549 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 138) took 2700 ms. Trace:
I20250110 02:51:13.618297 1181 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:13.618469 1181 ts_tablet_manager.cc:1397] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20250110 02:51:13.618671 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_38cc13d405674486842b52efad1baed0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.619117 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38cc13d405674486842b52efad1baed0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.619287 1181 raft_consensus.cc:357] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.619472 1181 raft_consensus.cc:383] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.619586 1181 raft_consensus.cc:738] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.619802 1181 consensus_queue.cc:260] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.621614 1181 ts_tablet_manager.cc:1428] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:13.621886 1181 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:13.616705 1180 rpcz_store.cc:276] 0110 02:51:10.916005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:10.916079 (+ 74us) service_pool.cc:225] Handling call
0110 02:51:13.616399 (+2700320us) ts_tablet_manager.cc:1145] Deleting tablet 0b74812cb81c44fcb8786cc5dae328e3
0110 02:51:13.616410 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.616515 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2700074}
I20250110 02:51:13.622815 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 163) took 2080 ms. Trace:
I20250110 02:51:13.622923 1180 rpcz_store.cc:276] 0110 02:51:11.542254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.542302 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:13.622651 (+2080349us) ts_tablet_manager.cc:1145] Deleting tablet 585320c0970c4097b195caaae33789b5
0110 02:51:13.622662 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.622775 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063113}
I20250110 02:51:13.625613 1181 tablet_bootstrap.cc:654] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.626639 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 164) took 2038 ms. Trace:
I20250110 02:51:13.626770 1180 rpcz_store.cc:276] 0110 02:51:11.587799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.587859 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:13.623125 (+2035266us) ts_tablet_manager.cc:1145] Deleting tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
0110 02:51:13.626442 (+ 3317us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.626568 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035093}
I20250110 02:51:13.632258 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a73fa23faec64b8f8178b0dac75d437a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.632895 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a73fa23faec64b8f8178b0dac75d437a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.635541 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 165) took 2038 ms. Trace:
I20250110 02:51:13.635661 1180 rpcz_store.cc:276] 0110 02:51:11.596918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.600337 (+ 3419us) service_pool.cc:225] Handling call
0110 02:51:13.627048 (+2026711us) ts_tablet_manager.cc:1145] Deleting tablet 9e72ed2d31a8467f8c092b48d593c1cc
0110 02:51:13.635376 (+ 8328us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.635506 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015375}
I20250110 02:51:13.637755 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2cb0f90ebf7248e1a9f43302db530a34" } new_table_name: "default.Table_daad63f955d54180b631be7ea9671a89" modify_external_catalogs: true
I20250110 02:51:13.638047 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 166) took 1987 ms. Trace:
I20250110 02:51:13.638162 1180 rpcz_store.cc:276] 0110 02:51:11.650689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.651417 (+ 728us) service_pool.cc:225] Handling call
0110 02:51:13.635897 (+1984480us) ts_tablet_manager.cc:1145] Deleting tablet 5af3d61366b2432db53b18f2b7bb49d9
0110 02:51:13.637886 (+ 1989us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.637995 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981096}
I20250110 02:51:13.638535 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 169) took 1918 ms. Trace:
I20250110 02:51:13.639015 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_4a33b1b80f7f40058628dfb357187113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.638656 1180 rpcz_store.cc:276] 0110 02:51:11.719976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.720678 (+ 702us) service_pool.cc:225] Handling call
0110 02:51:13.638396 (+1917718us) ts_tablet_manager.cc:1145] Deleting tablet 8940c201aa104b0b8bc8d866859fe8ab
0110 02:51:13.638404 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.638492 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1914092}
W20250110 02:51:13.639470 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a33b1b80f7f40058628dfb357187113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.639688 1180 tablet_replica.cc:331] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:13.639926 1180 raft_consensus.cc:2238] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.640149 1180 raft_consensus.cc:2267] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.640759 1180 ts_tablet_manager.cc:1905] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.641264 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 66f6f4f221e6445db0bb569f372cfaa0 (table default.Table_046451c051e2479b8b7de432ffb408bc [id=812b6d95174a4614857cae88273885cb]) successfully deleted
W20250110 02:51:13.641594 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 0b74812cb81c44fcb8786cc5dae328e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b74812cb81c44fcb8786cc5dae328e3 already in progress: creating tablet
W20250110 02:51:13.641779 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 585320c0970c4097b195caaae33789b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585320c0970c4097b195caaae33789b5 already in progress: creating tablet
W20250110 02:51:13.641961 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b already in progress: creating tablet
W20250110 02:51:13.642130 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9e72ed2d31a8467f8c092b48d593c1cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e72ed2d31a8467f8c092b48d593c1cc already in progress: creating tablet
W20250110 02:51:13.642280 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5af3d61366b2432db53b18f2b7bb49d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5af3d61366b2432db53b18f2b7bb49d9 already in progress: creating tablet
I20250110 02:51:13.642661 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 65285bce60fd4015b085deb7442eea6e (DEFAULT_TABLE table=default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:13.642804 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8940c201aa104b0b8bc8d866859fe8ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8940c201aa104b0b8bc8d866859fe8ab already in progress: creating tablet
I20250110 02:51:13.643078 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285bce60fd4015b085deb7442eea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.646708 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 (DEFAULT_TABLE table=default.Table_e8d197a297e647b5ad3b516ff8054704 [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.647119 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6555b0a4491046bf8f5cd4ee406c3955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.649694 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f9ee107db2ba4f2d9b62e9459ea41457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.650157 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9ee107db2ba4f2d9b62e9459ea41457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.650661 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ce4fe51f2e914865a2e8bbaa256b8098 (DEFAULT_TABLE table=default.Table_3b6669107e9943cc99dd6d48cd80c5a2 [id=7e19567637ca466c8df85f6e0aaee0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.650978 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fe51f2e914865a2e8bbaa256b8098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.654565 1030 tablet_service.cc:1467] Processing CreateTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 (DEFAULT_TABLE table=default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.654893 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb726d2c9a04963b92a3fb625cf84f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.655486 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98b789eaabe4191af40c9f713acb030" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.658495 881 tablet_service.cc:1467] Processing CreateTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 (DEFAULT_TABLE table=default.Table_e8d197a297e647b5ad3b516ff8054704 [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.658640 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 09fd9734714a4847ab2a4ebc37af8960 (DEFAULT_TABLE table=default.Table_627a9df6d7b84326bf7e90f5423ff267 [id=ee9fe75dbd374159bb7a40bdc3576d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.658691 898 tablet_service.cc:1467] Processing CreateTablet for tablet 65285bce60fd4015b085deb7442eea6e (DEFAULT_TABLE table=default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.658991 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fd9734714a4847ab2a4ebc37af8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.659446 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 197) took 1617 ms. Trace:
I20250110 02:51:13.661034 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa40d20288644b2a8d36ede967d16e79" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.658898 1188 raft_consensus.cc:357] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.662075 886 tablet_service.cc:1467] Processing CreateTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 (DEFAULT_TABLE table=default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.662585 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 (DEFAULT_TABLE table=default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.662614 895 tablet_service.cc:1467] Processing CreateTablet for tablet ce4fe51f2e914865a2e8bbaa256b8098 (DEFAULT_TABLE table=default.Table_3b6669107e9943cc99dd6d48cd80c5a2 [id=7e19567637ca466c8df85f6e0aaee0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.662858 890 tablet_service.cc:1467] Processing CreateTablet for tablet 09fd9734714a4847ab2a4ebc37af8960 (DEFAULT_TABLE table=default.Table_627a9df6d7b84326bf7e90f5423ff267 [id=ee9fe75dbd374159bb7a40bdc3576d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.663651 1178 rpcz_store.cc:276] 0110 02:51:12.037537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.037938 (+ 401us) service_pool.cc:225] Handling call
0110 02:51:13.588286 (+1550348us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:13.588294 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.654618 (+ 66324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534913}
I20250110 02:51:13.664062 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 198) took 1484 ms. Trace:
I20250110 02:51:13.664903 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285bce60fd4015b085deb7442eea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.665388 1188 raft_consensus.cc:383] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.664861 1178 rpcz_store.cc:276] 0110 02:51:12.179132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.179183 (+ 51us) service_pool.cc:225] Handling call
0110 02:51:13.663932 (+1484749us) ts_tablet_manager.cc:1145] Deleting tablet 3fa76357abae4a95b8fb3af47288e43a
0110 02:51:13.663940 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.664025 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484522}
I20250110 02:51:13.665616 879 tablet_service.cc:1467] Processing CreateTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 (DEFAULT_TABLE table=default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.665930 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 199) took 1474 ms. Trace:
I20250110 02:51:13.666250 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6555b0a4491046bf8f5cd4ee406c3955. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.666370 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7902775b8db247a89d9b20d0ee880e36 consensus state has no leader
I20250110 02:51:13.668725 1188 raft_consensus.cc:738] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.668942 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5d9b00106d6148598a57409f1df9ad37" } modify_external_catalogs: true
I20250110 02:51:13.669152 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5d9b00106d6148598a57409f1df9ad37" } modify_external_catalogs: true
I20250110 02:51:13.668064 520 catalog_manager.cc:5526] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:13.673209 520 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]). Expected version 2 got 0
I20250110 02:51:13.668505 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_83b75cdbc7424f579e0297ecd7b0d4a4" } modify_external_catalogs: true
I20250110 02:51:13.674018 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_83b75cdbc7424f579e0297ecd7b0d4a4" } modify_external_catalogs: true
I20250110 02:51:13.669044 1188 consensus_queue.cc:260] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.675300 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fe51f2e914865a2e8bbaa256b8098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.675334 918 raft_consensus.cc:2463] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:13.680033 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51adff7f50c4aec94e1892a50ec36c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.669854 1178 rpcz_store.cc:276] 0110 02:51:12.190975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.191027 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:13.665735 (+1474708us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:13.665749 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.665875 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2944,"tablet-delete.queue_time_us":1474489}
I20250110 02:51:13.681877 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb726d2c9a04963b92a3fb625cf84f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.682488 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51adff7f50c4aec94e1892a50ec36c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.682973 1188 ts_tablet_manager.cc:1428] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.096s user 0.001s sys 0.001s
I20250110 02:51:13.683243 1188 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:13.683614 1178 tablet_replica.cc:331] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:13.684060 1178 raft_consensus.cc:2238] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.684300 1178 raft_consensus.cc:2267] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.685040 1178 ts_tablet_manager.cc:1905] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.685396 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fd9734714a4847ab2a4ebc37af8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.686839 994 leader_election.cc:304] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:13.688848 994 leader_election.cc:304] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:13.688916 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2412420dc54258a6921ead194c1587 with delete_type TABLET_DATA_DELETED (replaced by tablet f14f06ba7d0f44418693e36f6bc6d888) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.689226 1467 raft_consensus.cc:2798] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:13.703442 1386 raft_consensus.cc:3054] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.703405 1467 raft_consensus.cc:491] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:13.704071 1467 raft_consensus.cc:3054] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.705802 1181 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:13.705971 1181 ts_tablet_manager.cc:1397] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.002s
W20250110 02:51:13.708184 510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd -> Peer 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025): Couldn't send request to peer 5d8d40556dc348e8aa8255d116251a10. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:39025: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250110 02:51:13.708425 1181 raft_consensus.cc:357] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.709218 1181 raft_consensus.cc:383] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.710220 1181 raft_consensus.cc:738] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.710491 1181 consensus_queue.cc:260] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.724102 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f40ec2d4223143dcb1175f357f06de68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.725283 1179 ts_tablet_manager.cc:1905] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:13.726126 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40ec2d4223143dcb1175f357f06de68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.726147 752 tablet_service.cc:1467] Processing CreateTablet for tablet 65285bce60fd4015b085deb7442eea6e (DEFAULT_TABLE table=default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.727046 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 (DEFAULT_TABLE table=default.Table_e8d197a297e647b5ad3b516ff8054704 [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.724831 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_123de8ace5ed47f7850040c5de4fcbc7" } modify_external_catalogs: true
I20250110 02:51:13.727377 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65285bce60fd4015b085deb7442eea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.727486 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_123de8ace5ed47f7850040c5de4fcbc7" } modify_external_catalogs: true
I20250110 02:51:13.727622 746 tablet_service.cc:1467] Processing CreateTablet for tablet ce4fe51f2e914865a2e8bbaa256b8098 (DEFAULT_TABLE table=default.Table_3b6669107e9943cc99dd6d48cd80c5a2 [id=7e19567637ca466c8df85f6e0aaee0bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.728005 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce4fe51f2e914865a2e8bbaa256b8098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.730614 744 tablet_service.cc:1467] Processing CreateTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 (DEFAULT_TABLE table=default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.731478 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb726d2c9a04963b92a3fb625cf84f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.731530 1179 ts_tablet_manager.cc:1918] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.731657 1179 log.cc:1198] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:13.732090 1179 ts_tablet_manager.cc:1939] T 70af6ec40c7a4442b91680d8125c2d48 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:13.733494 1179 tablet_replica.cc:331] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.733716 1179 raft_consensus.cc:2238] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.733922 1179 raft_consensus.cc:2267] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.734633 740 tablet_service.cc:1467] Processing CreateTablet for tablet 09fd9734714a4847ab2a4ebc37af8960 (DEFAULT_TABLE table=default.Table_627a9df6d7b84326bf7e90f5423ff267 [id=ee9fe75dbd374159bb7a40bdc3576d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.734993 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fd9734714a4847ab2a4ebc37af8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.736505 1179 ts_tablet_manager.cc:1905] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.738739 755 tablet_service.cc:1467] Processing CreateTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 (DEFAULT_TABLE table=default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.739181 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51adff7f50c4aec94e1892a50ec36c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.746199 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_61492b773a714042b0532061e607c827" } modify_external_catalogs: true
I20250110 02:51:13.746397 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_61492b773a714042b0532061e607c827" } modify_external_catalogs: true
I20250110 02:51:13.747083 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:13.747845 522 master_service.cc:946] ReplaceTablet: received request to replace tablet b375a932156d4abb908dbf327689b0f9 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.749429 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b43bb790e85343308a1bb1e1110bfb9c
W20250110 02:51:13.725193 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8b2297377bc4e5095826e30523eede7 consensus state has no leader
I20250110 02:51:13.751834 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:13.752830 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 70af6ec40c7a4442b91680d8125c2d48 (table default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]) successfully deleted
W20250110 02:51:13.753055 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 60e534bf66f044b88d8cda0f3a1228f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e534bf66f044b88d8cda0f3a1228f4 already in progress: creating tablet
W20250110 02:51:13.753237 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 already in progress: creating tablet
W20250110 02:51:13.753381 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b375a932156d4abb908dbf327689b0f9 already in progress: creating tablet
I20250110 02:51:13.763079 1180 ts_tablet_manager.cc:1918] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:13.766155 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f15078fc22f4b32a48c7190b36e2192 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.766462 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6555b0a4491046bf8f5cd4ee406c3955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.767146 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e25f9643ac0476a90c93fb9aeb921a3 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.770799 1184 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.770964 1184 ts_tablet_manager.cc:1397] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.198s user 0.009s sys 0.059s
I20250110 02:51:13.772759 1188 tablet_bootstrap.cc:654] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.773754 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.773944 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.774567 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.775277 1180 log.cc:1198] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/308f307f4200409aaa47df55f20472d0
I20250110 02:51:13.782586 1180 ts_tablet_manager.cc:1939] T 308f307f4200409aaa47df55f20472d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:13.783635 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 168) took 2063 ms. Trace:
I20250110 02:51:13.783726 1180 rpcz_store.cc:276] 0110 02:51:11.719886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.720759 (+ 873us) service_pool.cc:225] Handling call
0110 02:51:13.639592 (+1918833us) ts_tablet_manager.cc:1145] Deleting tablet 308f307f4200409aaa47df55f20472d0
0110 02:51:13.639603 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.644556 (+ 4953us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.709768 (+ 65212us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.783238 (+ 73470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1912409}
I20250110 02:51:13.784603 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 175) took 2008 ms. Trace:
I20250110 02:51:13.784691 1180 rpcz_store.cc:276] 0110 02:51:11.776546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.776822 (+ 276us) service_pool.cc:225] Handling call
0110 02:51:13.783943 (+2007121us) ts_tablet_manager.cc:1145] Deleting tablet 019c0d60da5548bb86c93ccd89bf94ff
0110 02:51:13.783950 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.784557 (+ 607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006869}
I20250110 02:51:13.784945 1180 tablet_replica.cc:331] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:13.785095 1180 raft_consensus.cc:2238] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:13.785347 1180 pending_rounds.cc:62] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:13.785401 1180 pending_rounds.cc:69] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611724002484224 op_type: NO_OP noop_request { }
I20250110 02:51:13.785539 1180 raft_consensus.cc:2883] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:13.785621 1180 raft_consensus.cc:2267] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.786099 1180 ts_tablet_manager.cc:1905] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.788945 1467 raft_consensus.cc:513] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.790038 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa40d20288644b2a8d36ede967d16e79" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:13.790935 918 raft_consensus.cc:3054] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.793274 918 raft_consensus.cc:2463] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:13.793576 1386 raft_consensus.cc:2743] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.796679 994 leader_election.cc:304] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:13.797418 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f051d394f83d4eb39d7d6a458a478718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.797766 1178 ts_tablet_manager.cc:1918] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.797895 1178 log.cc:1198] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/4c9d791b1c504171ba66dec55c8fb0e6
W20250110 02:51:13.798043 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f051d394f83d4eb39d7d6a458a478718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.798233 1178 ts_tablet_manager.cc:1939] T 4c9d791b1c504171ba66dec55c8fb0e6 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:13.798906 1386 raft_consensus.cc:2798] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:13.799055 1386 raft_consensus.cc:695] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:13.799742 1386 consensus_queue.cc:237] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.800230 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 200) took 1564 ms. Trace:
I20250110 02:51:13.800304 1178 rpcz_store.cc:276] 0110 02:51:12.235674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.235730 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:13.681331 (+1445601us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d791b1c504171ba66dec55c8fb0e6
0110 02:51:13.681341 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.774898 (+ 93557us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.797652 (+ 22754us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.800180 (+ 2528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445413}
I20250110 02:51:13.800676 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 207) took 1492 ms. Trace:
I20250110 02:51:13.800753 1178 rpcz_store.cc:276] 0110 02:51:12.308006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.310204 (+ 2198us) service_pool.cc:225] Handling call
0110 02:51:13.800529 (+1490325us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:13.800539 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.800632 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490129}
I20250110 02:51:13.801082 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 217) took 1431 ms. Trace:
I20250110 02:51:13.801160 1178 rpcz_store.cc:276] 0110 02:51:12.369264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.370885 (+ 1621us) service_pool.cc:225] Handling call
0110 02:51:13.800956 (+1430071us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:13.800963 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.801036 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429797}
I20250110 02:51:13.801450 1178 tablet_replica.cc:331] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:13.801607 1178 raft_consensus.cc:2238] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.801746 1178 raft_consensus.cc:2267] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.802229 1178 ts_tablet_manager.cc:1905] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.803961 1467 leader_election.cc:290] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.809303 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 308f307f4200409aaa47df55f20472d0 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
W20250110 02:51:13.809573 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 019c0d60da5548bb86c93ccd89bf94ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019c0d60da5548bb86c93ccd89bf94ff already in progress: creating tablet
I20250110 02:51:13.813397 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_04563a8a322645f7b582860698d0e791" } new_table_name: "default.Table_38807d5e51d64c94b251680c200dc98a" modify_external_catalogs: true
I20250110 02:51:13.814879 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3b6669107e9943cc99dd6d48cd80c5a2" } new_table_name: "default.Table_5f85d34e73f44a2a9eec8879fda3d338" modify_external_catalogs: true
I20250110 02:51:13.817338 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a1daaf2964134078b6e9ac86b7672a37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.817837 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1daaf2964134078b6e9ac86b7672a37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.819787 519 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]). Expected version 1 got 0
I20250110 02:51:13.820816 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0413ddbebc534e0da2f874ae0ecba335" } new_table_name: "default.Table_bc967f7066e24bb9b02c668d1f2b2be2" modify_external_catalogs: true
I20250110 02:51:13.827621 1188 tablet_bootstrap.cc:492] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:13.827801 1188 ts_tablet_manager.cc:1397] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.145s user 0.008s sys 0.002s
I20250110 02:51:13.829810 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 6479d71a5142436b8c6ea2f76c221b33 (DEFAULT_TABLE table=default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 [id=a09ade04ca6745f3899557ae0b416eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.830221 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6479d71a5142436b8c6ea2f76c221b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.830652 1037 tablet_service.cc:1467] Processing CreateTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.830684 1188 raft_consensus.cc:357] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.830886 1188 raft_consensus.cc:383] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.830960 1188 raft_consensus.cc:738] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.831107 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.831202 1188 consensus_queue.cc:260] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.834419 1188 ts_tablet_manager.cc:1428] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:13.834771 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca895eca14748c19fdb7d363499b544 (DEFAULT_TABLE table=default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.835143 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca895eca14748c19fdb7d363499b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.836329 1188 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:13.838332 1188 tablet_bootstrap.cc:654] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.838779 1041 tablet_service.cc:1467] Processing CreateTablet for tablet b12ce7a0dc4f4f3586a2f918d31234e1 (DEFAULT_TABLE table=default.Table_dd0bd0decd37446f891e12d29ab4955b [id=92d9235395b24e1eb75141de12dbe3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.839185 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12ce7a0dc4f4f3586a2f918d31234e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.844265 1427 raft_consensus.cc:491] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.846673 1178 ts_tablet_manager.cc:1918] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.846807 1178 log.cc:1198] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:13.847191 1178 ts_tablet_manager.cc:1939] T 2da6483de7f5475ea3d755a6efbda369 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:13.847322 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.847520 1314 raft_consensus.cc:491] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.847916 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 218) took 1464 ms. Trace:
I20250110 02:51:13.848582 1178 rpcz_store.cc:276] 0110 02:51:12.383125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.384487 (+ 1362us) service_pool.cc:225] Handling call
0110 02:51:13.801369 (+1416882us) ts_tablet_manager.cc:1145] Deleting tablet 2da6483de7f5475ea3d755a6efbda369
0110 02:51:13.801393 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.817973 (+ 16580us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.843080 (+ 25107us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.847869 (+ 4789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405163}
I20250110 02:51:13.850580 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 219) took 1401 ms. Trace:
I20250110 02:51:13.850710 1178 rpcz_store.cc:276] 0110 02:51:12.449112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.449198 (+ 86us) service_pool.cc:225] Handling call
0110 02:51:13.850444 (+1401246us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:13.850453 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.850551 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395770}
I20250110 02:51:13.851234 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 221) took 1401 ms. Trace:
I20250110 02:51:13.851337 1178 rpcz_store.cc:276] 0110 02:51:12.449796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.449849 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:13.851123 (+1401274us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:13.851131 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.851205 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395710}
I20250110 02:51:13.851372 1427 raft_consensus.cc:513] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.851926 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 46a4745356e3423f9d4892d242da1c48
I20250110 02:51:13.852083 1427 leader_election.cc:290] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.852126 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 220) took 1402 ms. Trace:
I20250110 02:51:13.852226 1427 raft_consensus.cc:491] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.852222 1178 rpcz_store.cc:276] 0110 02:51:12.449670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.453963 (+ 4293us) service_pool.cc:225] Handling call
0110 02:51:13.852027 (+1398064us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:13.852035 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.852101 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395345}
I20250110 02:51:13.852334 1427 raft_consensus.cc:513] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.852852 1427 leader_election.cc:290] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.853006 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.853219 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:13.853385 1314 raft_consensus.cc:513] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.853603 517 catalog_manager.cc:6596] ReplaceTablet: tablet 8f15078fc22f4b32a48c7190b36e2192 deleted and replaced by tablet bfd120d8edf14151b7ebed3b74e2b399
I20250110 02:51:13.853717 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dbb61a51f14a3f96cc3c77981e8ce5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.853938 1314 leader_election.cc:290] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:13.854048 749 tablet_service.cc:1467] Processing CreateTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.854300 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.854477 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:13.854629 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce4fe51f2e914865a2e8bbaa256b8098 consensus state has no leader
I20250110 02:51:13.854987 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:13.855475 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb20d47ace4fd49f9dc4e526cab702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:13.855705 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3352f8378b9846a3854086293f2d4e15 consensus state has no leader
I20250110 02:51:13.856825 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65704c9d486444ebe8e4ffd4580f395" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:13.858000 1178 tablet_replica.cc:331] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:13.858125 709 leader_election.cc:343] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.858327 709 leader_election.cc:343] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.859004 709 leader_election.cc:343] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:13.861644 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c4bd5e8317a4801a8134d58c23b7bee has no consensus state
I20250110 02:51:13.862516 753 tablet_service.cc:1467] Processing CreateTablet for tablet b12ce7a0dc4f4f3586a2f918d31234e1 (DEFAULT_TABLE table=default.Table_dd0bd0decd37446f891e12d29ab4955b [id=92d9235395b24e1eb75141de12dbe3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.862867 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12ce7a0dc4f4f3586a2f918d31234e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.863363 754 tablet_service.cc:1467] Processing CreateTablet for tablet 6479d71a5142436b8c6ea2f76c221b33 (DEFAULT_TABLE table=default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 [id=a09ade04ca6745f3899557ae0b416eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.863826 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.863898 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca895eca14748c19fdb7d363499b544 (DEFAULT_TABLE table=default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.864212 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6479d71a5142436b8c6ea2f76c221b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.864611 1184 raft_consensus.cc:357] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.864801 1184 raft_consensus.cc:383] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.865312 1184 raft_consensus.cc:738] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.865557 1184 consensus_queue.cc:260] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.866887 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.868381 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:13.869613 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.869807 1178 raft_consensus.cc:2238] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.870024 1178 raft_consensus.cc:2267] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.870589 1178 ts_tablet_manager.cc:1905] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.875687 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca895eca14748c19fdb7d363499b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.875556 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_18a8f14e969a458599b59ee194888979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.875717 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.875716 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_627a9df6d7b84326bf7e90f5423ff267" } new_table_name: "default.Table_f8df7cc638174115bd8c014aede44d8d" modify_external_catalogs: true
W20250110 02:51:13.876121 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18a8f14e969a458599b59ee194888979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.877890 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:13.879520 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8cf4c26686b84eb59c11d17010f8a679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.880265 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3ae68e4f0254ef38aafd5a1f8f88a54 already in progress: creating tablet
W20250110 02:51:13.880513 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3fa76357abae4a95b8fb3af47288e43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa76357abae4a95b8fb3af47288e43a already in progress: creating tablet
W20250110 02:51:13.880694 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d9018955f3884824beb2e7336987fc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9018955f3884824beb2e7336987fc1c already in progress: creating tablet
W20250110 02:51:13.880270 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cf4c26686b84eb59c11d17010f8a679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.881312 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.881541 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3f60c819f4824bc99d755e7442976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:09 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.881554 1179 ts_tablet_manager.cc:1918] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.881702 1179 log.cc:1198] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/514704cd935b44e8b760221224332509
I20250110 02:51:13.881716 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.881922 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.881969 1184 ts_tablet_manager.cc:1428] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.095s user 0.000s sys 0.002s
I20250110 02:51:13.882110 898 tablet_service.cc:1514] Processing DeleteTablet for tablet aa40d20288644b2a8d36ede967d16e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 60e534bf66f044b88d8cda0f3a1228f4) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.882469 881 tablet_service.cc:1467] Processing CreateTablet for tablet b12ce7a0dc4f4f3586a2f918d31234e1 (DEFAULT_TABLE table=default.Table_dd0bd0decd37446f891e12d29ab4955b [id=92d9235395b24e1eb75141de12dbe3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.882694 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 4c9d791b1c504171ba66dec55c8fb0e6 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:13.882823 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b12ce7a0dc4f4f3586a2f918d31234e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.882913 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2ef72e05f8ed4d46b759c7a04497653b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ef72e05f8ed4d46b759c7a04497653b already in progress: creating tablet
I20250110 02:51:13.883038 1179 ts_tablet_manager.cc:1939] T 514704cd935b44e8b760221224332509 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:13.883111 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5e183122d13a4c38988bf2fe6f7a3694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e183122d13a4c38988bf2fe6f7a3694 already in progress: creating tablet
I20250110 02:51:13.883291 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2da6483de7f5475ea3d755a6efbda369 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
W20250110 02:51:13.883476 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7d4422eca5604d149e73c2a669d07af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d4422eca5604d149e73c2a669d07af0 already in progress: creating tablet
W20250110 02:51:13.883647 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet dad0d561a8ef498ba13bc813d4cd82fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dad0d561a8ef498ba13bc813d4cd82fa already in progress: creating tablet
W20250110 02:51:13.883841 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fc9826ef554a5699e365d10d02e97a already in progress: creating tablet
I20250110 02:51:13.884109 1179 tablet_replica.cc:331] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.887063 1184 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:13.887099 882 tablet_service.cc:1467] Processing CreateTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.887539 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.888693 892 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca895eca14748c19fdb7d363499b544 (DEFAULT_TABLE table=default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.889138 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca895eca14748c19fdb7d363499b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.890007 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6479d71a5142436b8c6ea2f76c221b33 (DEFAULT_TABLE table=default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 [id=a09ade04ca6745f3899557ae0b416eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.890333 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6479d71a5142436b8c6ea2f76c221b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.890877 1180 ts_tablet_manager.cc:1918] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:13.891132 1180 log.cc:1198] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/67adef4c3f4041e1ab480100ca762f62
I20250110 02:51:13.890506 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.891547 1180 ts_tablet_manager.cc:1939] T 67adef4c3f4041e1ab480100ca762f62 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:13.890657 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.890769 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:13.892863 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 130268us
I20250110 02:51:13.893209 1179 raft_consensus.cc:2238] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.893379 1179 raft_consensus.cc:2267] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.893880 1179 ts_tablet_manager.cc:1905] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:13.895864 996 leader_election.cc:343] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: aa40d20288644b2a8d36ede967d16e79
I20250110 02:51:13.900951 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e10e7654cec495bb532fb00e73c020d from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:13.901731 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65704c9d486444ebe8e4ffd4580f395" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:13.902397 711 leader_election.cc:343] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.902570 711 leader_election.cc:304] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.903200 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dbb61a51f14a3f96cc3c77981e8ce5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.903462 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb20d47ace4fd49f9dc4e526cab702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.903867 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 167) took 2181 ms. Trace:
I20250110 02:51:13.903999 1180 rpcz_store.cc:276] 0110 02:51:11.719538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.720985 (+ 1447us) service_pool.cc:225] Handling call
0110 02:51:13.784872 (+2063887us) ts_tablet_manager.cc:1145] Deleting tablet 67adef4c3f4041e1ab480100ca762f62
0110 02:51:13.784880 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.857026 (+ 72146us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.890774 (+ 33748us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.900079 (+ 9305us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1976680}
W20250110 02:51:13.904209 711 leader_election.cc:343] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.904376 711 leader_election.cc:304] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:13.904582 711 raft_consensus.cc:2681] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 02:51:13.904938 711 leader_election.cc:343] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:13.905128 711 leader_election.cc:304] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:13.905505 1314 raft_consensus.cc:2743] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.905803 1512 raft_consensus.cc:2743] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.906386 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 177) took 2111 ms. Trace:
I20250110 02:51:13.906478 1180 rpcz_store.cc:276] 0110 02:51:11.792978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.808217 (+ 15239us) service_pool.cc:225] Handling call
0110 02:51:13.904428 (+2096211us) ts_tablet_manager.cc:1145] Deleting tablet f14f06ba7d0f44418693e36f6bc6d888
0110 02:51:13.904438 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.904516 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2086767}
I20250110 02:51:13.908318 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 183) took 2062 ms. Trace:
I20250110 02:51:13.908455 1180 rpcz_store.cc:276] 0110 02:51:11.844857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.904916 (+ 60059us) service_pool.cc:225] Handling call
0110 02:51:13.906832 (+2001916us) ts_tablet_manager.cc:1145] Deleting tablet 48cb20d47ace4fd49f9dc4e526cab702
0110 02:51:13.906841 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.906939 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999362}
I20250110 02:51:13.909114 1184 tablet_bootstrap.cc:654] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.910158 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 192) took 2012 ms. Trace:
I20250110 02:51:13.910247 1180 rpcz_store.cc:276] 0110 02:51:11.896684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:11.925913 (+ 29229us) service_pool.cc:225] Handling call
0110 02:51:13.908749 (+1982836us) ts_tablet_manager.cc:1145] Deleting tablet 64c77ec8c7724d5fb408a8bfa53811eb
0110 02:51:13.908757 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.908835 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982596}
I20250110 02:51:13.916383 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 198) took 1897 ms. Trace:
I20250110 02:51:13.916484 1180 rpcz_store.cc:276] 0110 02:51:12.017131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.017358 (+ 227us) service_pool.cc:225] Handling call
0110 02:51:13.914504 (+1897146us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:13.914514 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.914606 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896902}
I20250110 02:51:13.918205 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 195) took 1910 ms. Trace:
I20250110 02:51:13.918294 1180 rpcz_store.cc:276] 0110 02:51:12.006275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.006342 (+ 67us) service_pool.cc:225] Handling call
0110 02:51:13.916803 (+1910461us) ts_tablet_manager.cc:1145] Deleting tablet f9dbb61a51f14a3f96cc3c77981e8ce5
0110 02:51:13.916812 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.916897 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886117}
I20250110 02:51:13.919550 1180 tablet_replica.cc:331] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:13.926821 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_73259c80bde24fd3a2440bdce73bd88e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.927431 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73259c80bde24fd3a2440bdce73bd88e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.928769 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_105ea910a3c447a6abb0c0919ab8ee06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.929225 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_105ea910a3c447a6abb0c0919ab8ee06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.929497 1178 ts_tablet_manager.cc:1918] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.929634 1178 log.cc:1198] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/10ab917bde534e3fbdf5dbae4af9304e
I20250110 02:51:13.930136 1178 ts_tablet_manager.cc:1939] T 10ab917bde534e3fbdf5dbae4af9304e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:13.930924 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 222) took 1273 ms. Trace:
I20250110 02:51:13.931008 1178 rpcz_store.cc:276] 0110 02:51:12.657160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.661474 (+ 4314us) service_pool.cc:225] Handling call
0110 02:51:13.857927 (+1196453us) ts_tablet_manager.cc:1145] Deleting tablet 10ab917bde534e3fbdf5dbae4af9304e
0110 02:51:13.857937 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.874758 (+ 16821us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.929374 (+ 54616us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.930858 (+ 1484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189163}
I20250110 02:51:13.931473 1181 ts_tablet_manager.cc:1428] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.225s user 0.009s sys 0.005s
I20250110 02:51:13.931536 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 224) took 1208 ms. Trace:
I20250110 02:51:13.931797 1178 rpcz_store.cc:276] 0110 02:51:12.722905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.722964 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:13.931371 (+1208407us) ts_tablet_manager.cc:1145] Deleting tablet 1602ade49b7442d6b8e6f4b85d2e80a1
0110 02:51:13.931379 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.931489 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208129}
I20250110 02:51:13.932301 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 225) took 1208 ms. Trace:
I20250110 02:51:13.932380 1178 rpcz_store.cc:276] 0110 02:51:12.723394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.723438 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:13.932158 (+1208720us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:13.932166 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.932251 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208522}
I20250110 02:51:13.932518 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 67adef4c3f4041e1ab480100ca762f62 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
W20250110 02:51:13.932782 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f14f06ba7d0f44418693e36f6bc6d888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14f06ba7d0f44418693e36f6bc6d888 already in progress: creating tablet
I20250110 02:51:13.932811 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 226) took 1209 ms. Trace:
W20250110 02:51:13.932984 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 48cb20d47ace4fd49f9dc4e526cab702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48cb20d47ace4fd49f9dc4e526cab702 already in progress: creating tablet
W20250110 02:51:13.933004 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09fd9734714a4847ab2a4ebc37af8960 consensus state has no leader
W20250110 02:51:13.933194 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 64c77ec8c7724d5fb408a8bfa53811eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64c77ec8c7724d5fb408a8bfa53811eb already in progress: creating tablet
W20250110 02:51:13.933378 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3ae68e4f0254ef38aafd5a1f8f88a54 already in progress: creating tablet
W20250110 02:51:13.933568 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9dbb61a51f14a3f96cc3c77981e8ce5 already in progress: creating tablet
I20250110 02:51:13.933501 1178 rpcz_store.cc:276] 0110 02:51:12.723796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.723858 (+ 62us) service_pool.cc:225] Handling call
0110 02:51:13.932681 (+1208823us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:13.932688 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.932761 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208676}
I20250110 02:51:13.933827 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 514704cd935b44e8b760221224332509 (table default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]) successfully deleted
I20250110 02:51:13.933949 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 227) took 1209 ms. Trace:
W20250110 02:51:13.934032 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e744cdc1fcb94d46b48b684763ecf1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e744cdc1fcb94d46b48b684763ecf1bf already in progress: creating tablet
I20250110 02:51:13.934015 1178 rpcz_store.cc:276] 0110 02:51:12.724156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.724194 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:13.933802 (+1209608us) ts_tablet_manager.cc:1145] Deleting tablet 344184e1c98d422095cb254f5b2bc792
0110 02:51:13.933813 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.933896 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209459}
I20250110 02:51:13.934420 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 228) took 1209 ms. Trace:
I20250110 02:51:13.934585 1188 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:13.934501 1178 rpcz_store.cc:276] 0110 02:51:12.724439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.724474 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:13.934261 (+1209787us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:13.934268 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.934347 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209647}
I20250110 02:51:13.934726 1188 ts_tablet_manager.cc:1397] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.000s
I20250110 02:51:13.935467 1188 raft_consensus.cc:357] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.935635 1188 raft_consensus.cc:383] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.935684 1181 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:13.935729 1188 raft_consensus.cc:738] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.935942 1188 consensus_queue.cc:260] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.936394 1188 ts_tablet_manager.cc:1428] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:13.936681 1188 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:13.938325 1188 tablet_bootstrap.cc:654] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.938894 1181 tablet_bootstrap.cc:654] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:13.940698 1179 ts_tablet_manager.cc:1918] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.940840 1179 log.cc:1198] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f9dbb61a51f14a3f96cc3c77981e8ce5
I20250110 02:51:13.941092 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 10ab917bde534e3fbdf5dbae4af9304e (table default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]) successfully deleted
W20250110 02:51:13.941481 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d20177229f7b4f11875e0007afb1dd00 already in progress: creating tablet
W20250110 02:51:13.941673 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcc9a4a842914a6f878d01194cd9a5f7 already in progress: creating tablet
I20250110 02:51:13.941782 1179 ts_tablet_manager.cc:1939] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:13.941888 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 344184e1c98d422095cb254f5b2bc792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344184e1c98d422095cb254f5b2bc792 already in progress: creating tablet
W20250110 02:51:13.942116 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f4778347f054b86af697dbe3b287263 already in progress: creating tablet
I20250110 02:51:13.942771 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:13.944355 1180 raft_consensus.cc:2238] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.944603 1180 raft_consensus.cc:2267] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.945079 1180 ts_tablet_manager.cc:1905] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.945741 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 229) took 1186 ms. Trace:
I20250110 02:51:13.945874 1178 rpcz_store.cc:276] 0110 02:51:12.758895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.758943 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:13.945080 (+1186137us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:13.945091 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.945692 (+ 601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185932}
I20250110 02:51:13.948369 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:13.948524 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 241) took 1117 ms. Trace:
I20250110 02:51:13.948658 1178 rpcz_store.cc:276] 0110 02:51:12.831154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.831196 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:13.946317 (+1115121us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:13.946328 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.948417 (+ 2089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114940}
I20250110 02:51:13.949043 1178 tablet_replica.cc:331] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:13.949242 1178 raft_consensus.cc:2238] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.949448 1178 raft_consensus.cc:2267] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.951449 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0059fbcbfb7a4f239a5947f702d21c0f (DEFAULT_TABLE table=default.Table_38cc13d405674486842b52efad1baed0 [id=cba096e296394b3ca68bc08f9770efcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.951958 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0059fbcbfb7a4f239a5947f702d21c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.952178 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (DEFAULT_TABLE table=default.Table_a73fa23faec64b8f8178b0dac75d437a [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.952522 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00698a0a9ee4e5da69a745f4a3aa8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.955145 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 7efa857721c34003ad91912ef2d7b381 (DEFAULT_TABLE table=default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.955312 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 80186dd7d1a240068584b64ce9e9b03f (DEFAULT_TABLE table=default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.955471 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efa857721c34003ad91912ef2d7b381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.955724 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80186dd7d1a240068584b64ce9e9b03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.956897 1184 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:13.957058 1184 ts_tablet_manager.cc:1397] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.004s
I20250110 02:51:13.957252 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3e52914a424cb3a8bf422b3a56fb22 (DEFAULT_TABLE table=default.Table_f9ee107db2ba4f2d9b62e9459ea41457 [id=0e2933f55f834e8baa0a11bda1b4acb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.957643 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3e52914a424cb3a8bf422b3a56fb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.957664 1178 ts_tablet_manager.cc:1905] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.957875 1184 raft_consensus.cc:357] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.958099 1184 raft_consensus.cc:383] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.958168 1184 raft_consensus.cc:738] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.958667 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4bd5e8317a4801a8134d58c23b7bee (DEFAULT_TABLE table=default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.958878 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4bd5e8317a4801a8134d58c23b7bee (DEFAULT_TABLE table=default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.959014 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4bd5e8317a4801a8134d58c23b7bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.959249 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4bd5e8317a4801a8134d58c23b7bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.959712 749 tablet_service.cc:1467] Processing CreateTablet for tablet 80186dd7d1a240068584b64ce9e9b03f (DEFAULT_TABLE table=default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.960043 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80186dd7d1a240068584b64ce9e9b03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.960367 745 tablet_service.cc:1467] Processing CreateTablet for tablet 7efa857721c34003ad91912ef2d7b381 (DEFAULT_TABLE table=default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.960650 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efa857721c34003ad91912ef2d7b381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.961270 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0059fbcbfb7a4f239a5947f702d21c0f (DEFAULT_TABLE table=default.Table_38cc13d405674486842b52efad1baed0 [id=cba096e296394b3ca68bc08f9770efcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.961609 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0059fbcbfb7a4f239a5947f702d21c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.962756 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (table default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]) successfully deleted
I20250110 02:51:13.963726 1178 ts_tablet_manager.cc:1918] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.963842 1178 log.cc:1198] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/0c2412420dc54258a6921ead194c1587
I20250110 02:51:13.963847 1179 tablet_replica.cc:331] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:13.964035 1179 raft_consensus.cc:2238] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:13.964213 1178 ts_tablet_manager.cc:1939] T 0c2412420dc54258a6921ead194c1587 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:13.964212 1179 raft_consensus.cc:2267] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:13.964699 1179 ts_tablet_manager.cc:1905] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:13.964939 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 239) took 1133 ms. Trace:
I20250110 02:51:13.965569 1178 rpcz_store.cc:276] 0110 02:51:12.830958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.857406 (+ 26448us) service_pool.cc:225] Handling call
0110 02:51:13.948978 (+1091572us) ts_tablet_manager.cc:1145] Deleting tablet 0c2412420dc54258a6921ead194c1587
0110 02:51:13.948987 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.960947 (+ 11960us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.963635 (+ 2688us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.964894 (+ 1259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091142}
I20250110 02:51:13.965945 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_befac4c75a404c7aa6d15f1fac47ce8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:13.966588 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_86a6941cdab84e57989003542ba3eb76" } modify_external_catalogs: true
I20250110 02:51:13.966671 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b7a2bc15b1114d39944fadd733f2eb4d" } modify_external_catalogs: true
I20250110 02:51:13.966867 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b7a2bc15b1114d39944fadd733f2eb4d" } modify_external_catalogs: true
I20250110 02:51:13.967001 1367 raft_consensus.cc:491] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:13.967146 1367 raft_consensus.cc:513] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:13.967928 1367 leader_election.cc:290] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:13.967917 522 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:13.968240 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_39db324aabe548a28775a241e962cfe7" } new_table_name: "default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7" modify_external_catalogs: true
W20250110 02:51:13.968369 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_befac4c75a404c7aa6d15f1fac47ce8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.968432 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3f60c819f4824bc99d755e7442976" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:13.968703 1059 raft_consensus.cc:2388] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:13.969411 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_add5accf5d07401eadb673123fda5ae6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.969851 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_add5accf5d07401eadb673123fda5ae6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.966873 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_86a6941cdab84e57989003542ba3eb76" } modify_external_catalogs: true
I20250110 02:51:13.970934 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_87bb664f5b24476b9dd295e4fb7b7f20" } modify_external_catalogs: true
I20250110 02:51:13.971079 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_87bb664f5b24476b9dd295e4fb7b7f20" } modify_external_catalogs: true
W20250110 02:51:13.972208 853 leader_election.cc:343] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 02d3f60c819f4824bc99d755e7442976
I20250110 02:51:13.972327 853 leader_election.cc:304] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:13.972849 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_70a31a6f02a443d68091578355b71adc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.973337 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70a31a6f02a443d68091578355b71adc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.974776 1367 raft_consensus.cc:3054] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:13.976228 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 240) took 1145 ms. Trace:
I20250110 02:51:13.976341 1178 rpcz_store.cc:276] 0110 02:51:12.831058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.857621 (+ 26563us) service_pool.cc:225] Handling call
0110 02:51:13.976056 (+1118435us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:13.976067 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.976183 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118004}
I20250110 02:51:13.977149 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 250) took 1063 ms. Trace:
I20250110 02:51:13.977314 1178 rpcz_store.cc:276] 0110 02:51:12.912924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.912971 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:13.976579 (+1063608us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:13.976588 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.976663 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1063391}
I20250110 02:51:13.977694 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 248) took 1065 ms. Trace:
I20250110 02:51:13.977789 1178 rpcz_store.cc:276] 0110 02:51:12.912695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.913234 (+ 539us) service_pool.cc:225] Handling call
0110 02:51:13.977581 (+1064347us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:13.977589 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.977661 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064199}
I20250110 02:51:13.978130 1367 raft_consensus.cc:2743] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:13.978152 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 253) took 1045 ms. Trace:
I20250110 02:51:13.978281 1178 rpcz_store.cc:276] 0110 02:51:12.932601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.932648 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:13.978022 (+1045374us) ts_tablet_manager.cc:1145] Deleting tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8
0110 02:51:13.978029 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.978119 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045154}
I20250110 02:51:13.978672 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 256) took 1011 ms. Trace:
I20250110 02:51:13.978771 1178 rpcz_store.cc:276] 0110 02:51:12.966809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.966859 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:13.978527 (+1011668us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:13.978534 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.978630 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011432}
W20250110 02:51:13.979593 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f5e88d1bbc304b4ebbc849a7d783052e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e88d1bbc304b4ebbc849a7d783052e already in progress: creating tablet
W20250110 02:51:13.979838 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d5ddbc6b02742609bc2ca84c863a9d0 already in progress: creating tablet
I20250110 02:51:13.979984 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3e52914a424cb3a8bf422b3a56fb22 (DEFAULT_TABLE table=default.Table_f9ee107db2ba4f2d9b62e9459ea41457 [id=0e2933f55f834e8baa0a11bda1b4acb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.980036 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 0c2412420dc54258a6921ead194c1587 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
W20250110 02:51:13.980211 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b6d57f3e9a44d178300d3505d504854 already in progress: creating tablet
I20250110 02:51:13.980331 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3e52914a424cb3a8bf422b3a56fb22. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.980412 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff9b10c419242cf85ef1b708bd5a862 already in progress: creating tablet
W20250110 02:51:13.980595 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f66603a0ee8c4082807c5a6aabe7e339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f66603a0ee8c4082807c5a6aabe7e339 already in progress: creating tablet
W20250110 02:51:13.980785 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 already in progress: creating tablet
W20250110 02:51:13.980983 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 0127de45264f4660b843734a530eb310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0127de45264f4660b843734a530eb310 already in progress: creating tablet
I20250110 02:51:13.982633 882 tablet_service.cc:1467] Processing CreateTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (DEFAULT_TABLE table=default.Table_a73fa23faec64b8f8178b0dac75d437a [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.982972 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00698a0a9ee4e5da69a745f4a3aa8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.983122 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3375784ee4514b079aaf4ef21656d9ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:13.983589 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3375784ee4514b079aaf4ef21656d9ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:13.986634 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7efa857721c34003ad91912ef2d7b381 (DEFAULT_TABLE table=default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.986995 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efa857721c34003ad91912ef2d7b381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.990638 887 tablet_service.cc:1467] Processing CreateTablet for tablet 0059fbcbfb7a4f239a5947f702d21c0f (DEFAULT_TABLE table=default.Table_38cc13d405674486842b52efad1baed0 [id=cba096e296394b3ca68bc08f9770efcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.990933 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0059fbcbfb7a4f239a5947f702d21c0f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:13.963347 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a81c031451104d5a93a1672b4b1918c5 already in progress: creating tablet
I20250110 02:51:13.994597 888 tablet_service.cc:1467] Processing CreateTablet for tablet 80186dd7d1a240068584b64ce9e9b03f (DEFAULT_TABLE table=default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:13.995105 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 27a7885a80414f2193400672aa16a911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27a7885a80414f2193400672aa16a911 already in progress: creating tablet
I20250110 02:51:13.995409 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80186dd7d1a240068584b64ce9e9b03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.996158 1181 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:13.996301 1181 ts_tablet_manager.cc:1397] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20250110 02:51:13.996726 1180 ts_tablet_manager.cc:1918] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:13.996840 1180 log.cc:1198] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/10ab917bde534e3fbdf5dbae4af9304e
I20250110 02:51:13.997238 1180 ts_tablet_manager.cc:1939] T 10ab917bde534e3fbdf5dbae4af9304e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:13.997123 1181 raft_consensus.cc:357] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.997339 1181 raft_consensus.cc:383] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:13.997411 1181 raft_consensus.cc:738] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:13.997671 1181 consensus_queue.cc:260] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:13.998220 1181 ts_tablet_manager.cc:1428] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:13.998500 894 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4bd5e8317a4801a8134d58c23b7bee (DEFAULT_TABLE table=default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.998833 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4bd5e8317a4801a8134d58c23b7bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:13.973774 753 tablet_service.cc:1467] Processing CreateTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (DEFAULT_TABLE table=default.Table_a73fa23faec64b8f8178b0dac75d437a [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.000345 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d00698a0a9ee4e5da69a745f4a3aa8b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.002422 1181 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:13.974165 754 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3e52914a424cb3a8bf422b3a56fb22 (DEFAULT_TABLE table=default.Table_f9ee107db2ba4f2d9b62e9459ea41457 [id=0e2933f55f834e8baa0a11bda1b4acb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:13.990543 1184 consensus_queue.cc:260] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.005640 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3e52914a424cb3a8bf422b3a56fb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.005807 1184 ts_tablet_manager.cc:1428] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250110 02:51:14.006114 1184 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.009397 1184 tablet_bootstrap.cc:654] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.013772 896 tablet_service.cc:1467] Processing CreateTablet for tablet e03d53860ab744968693cf96467c3a19 (DEFAULT_TABLE table=default.Table_3a64d77e95f7457490d8a6641411e1dc [id=2983e0e083914a099f399cf16f1c30ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.014137 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03d53860ab744968693cf96467c3a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.014885 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2e10e7654cec495bb532fb00e73c020d
I20250110 02:51:14.015286 520 catalog_manager.cc:6596] ReplaceTablet: tablet 2e10e7654cec495bb532fb00e73c020d deleted and replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b
I20250110 02:51:14.016850 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 8e79175853474cbea951f5a5686bf630 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.017500 755 tablet_service.cc:1467] Processing CreateTablet for tablet c1608d90ceb64602889b5705ba4ab55d (DEFAULT_TABLE table=default.Table_4a33b1b80f7f40058628dfb357187113 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.017983 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1608d90ceb64602889b5705ba4ab55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.018783 898 tablet_service.cc:1467] Processing CreateTablet for tablet c1608d90ceb64602889b5705ba4ab55d (DEFAULT_TABLE table=default.Table_4a33b1b80f7f40058628dfb357187113 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.018927 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 2e25f9643ac0476a90c93fb9aeb921a3
I20250110 02:51:14.019098 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1608d90ceb64602889b5705ba4ab55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.019308 525 catalog_manager.cc:6596] ReplaceTablet: tablet 2e25f9643ac0476a90c93fb9aeb921a3 deleted and replaced by tablet d637597047ab4738a17eca2f5409459a
I20250110 02:51:14.022898 754 tablet_service.cc:1467] Processing CreateTablet for tablet e03d53860ab744968693cf96467c3a19 (DEFAULT_TABLE table=default.Table_3a64d77e95f7457490d8a6641411e1dc [id=2983e0e083914a099f399cf16f1c30ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.023277 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03d53860ab744968693cf96467c3a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.024104 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 200) took 1936 ms. Trace:
I20250110 02:51:14.024605 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.024818 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.024861 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e03d53860ab744968693cf96467c3a19 (DEFAULT_TABLE table=default.Table_3a64d77e95f7457490d8a6641411e1dc [id=2983e0e083914a099f399cf16f1c30ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.028297 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.028442 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.028512 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.028681 1180 rpcz_store.cc:276] 0110 02:51:12.063397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.064398 (+ 1001us) service_pool.cc:225] Handling call
0110 02:51:13.919477 (+1855079us) ts_tablet_manager.cc:1145] Deleting tablet 10ab917bde534e3fbdf5dbae4af9304e
0110 02:51:13.919486 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:13.948092 (+ 28606us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:13.996599 (+ 48507us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.000021 (+ 3422us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852091}
I20250110 02:51:14.029047 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 201) took 1848 ms. Trace:
I20250110 02:51:14.029312 1180 rpcz_store.cc:276] 0110 02:51:12.180311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.180368 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:14.028919 (+1848551us) ts_tablet_manager.cc:1145] Deleting tablet 3fa76357abae4a95b8fb3af47288e43a
0110 02:51:14.028927 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.029009 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848336}
I20250110 02:51:14.029664 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 202) took 1845 ms. Trace:
I20250110 02:51:14.029983 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 10ab917bde534e3fbdf5dbae4af9304e (table default.Table_a1233829553f48db897d4e93aa37c38d [id=fb9ffcd2425f470b8d98af429c256602]) successfully deleted
W20250110 02:51:14.030234 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3fa76357abae4a95b8fb3af47288e43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fa76357abae4a95b8fb3af47288e43a already in progress: creating tablet
I20250110 02:51:14.030364 1180 rpcz_store.cc:276] 0110 02:51:12.184064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.185409 (+ 1345us) service_pool.cc:225] Handling call
0110 02:51:14.029523 (+1844114us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:14.029532 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.029627 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1843760}
I20250110 02:51:14.030633 1180 tablet_replica.cc:331] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.030954 1180 raft_consensus.cc:2238] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:14.031177 1180 pending_rounds.cc:62] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:14.031452 1180 pending_rounds.cc:69] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611729517707264 op_type: NO_OP noop_request { }
I20250110 02:51:14.031576 1180 raft_consensus.cc:2883] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:14.031695 1188 tablet_bootstrap.cc:492] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.031831 1188 ts_tablet_manager.cc:1397] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.000s
I20250110 02:51:14.032564 1188 raft_consensus.cc:357] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.032759 1188 raft_consensus.cc:383] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.032825 1188 raft_consensus.cc:738] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.033041 1188 consensus_queue.cc:260] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.033699 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.033788 1188 ts_tablet_manager.cc:1428] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:14.034058 1188 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:14.035241 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d9018955f3884824beb2e7336987fc1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9018955f3884824beb2e7336987fc1c already in progress: creating tablet
I20250110 02:51:14.035876 1188 tablet_bootstrap.cc:654] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.036244 1181 tablet_bootstrap.cc:654] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.037408 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.042927 1180 raft_consensus.cc:2267] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.043949 1180 ts_tablet_manager.cc:1905] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.045173 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f6df074974974d81a83420e7a3a78070
W20250110 02:51:14.046001 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 598bc151d0c14fc6bb28523c535d9366 consensus state has no leader
I20250110 02:51:14.046358 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:14.046867 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:14.047430 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8e79175853474cbea951f5a5686bf630
I20250110 02:51:14.047839 520 catalog_manager.cc:6596] ReplaceTablet: tablet 8e79175853474cbea951f5a5686bf630 deleted and replaced by tablet 08705d260c0a462b94ba6b0dc5121123
I20250110 02:51:14.048775 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.048935 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c1608d90ceb64602889b5705ba4ab55d (DEFAULT_TABLE table=default.Table_4a33b1b80f7f40058628dfb357187113 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.049093 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03d53860ab744968693cf96467c3a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.049630 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6a8868f4b22043c193efdd181759b4f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.050104 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a8868f4b22043c193efdd181759b4f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.053452 1179 ts_tablet_manager.cc:1918] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.053584 1179 log.cc:1198] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/48cb20d47ace4fd49f9dc4e526cab702
I20250110 02:51:14.053984 1179 ts_tablet_manager.cc:1939] T 48cb20d47ace4fd49f9dc4e526cab702 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.059773 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b74812cb81c44fcb8786cc5dae328e3 with delete_type TABLET_DATA_DELETED (replaced by tablet e8b2297377bc4e5095826e30523eede7) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.060240 1178 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33070 (request call id 257) took 1088 ms. Trace:
I20250110 02:51:14.060352 1178 rpcz_store.cc:276] 0110 02:51:12.971593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.971641 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:14.060097 (+1088456us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:14.060107 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.060198 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088243}
I20250110 02:51:14.061250 1179 tablet_replica.cc:331] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.061450 1179 raft_consensus.cc:2238] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.061582 1179 raft_consensus.cc:2267] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.062649 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1608d90ceb64602889b5705ba4ab55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.063253 1179 ts_tablet_manager.cc:1905] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.068352 1178 tablet_replica.cc:331] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.068537 1178 raft_consensus.cc:2238] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.068686 1178 raft_consensus.cc:2267] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.069180 1179 ts_tablet_manager.cc:1918] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.069196 1178 ts_tablet_manager.cc:1905] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.069157 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8b4cf44baefc4dda95dd9dc4d17dcd9e" } new_table_name: "default.Table_658b8237dabd45f1a02c1fb92070f6e0" modify_external_catalogs: true
I20250110 02:51:14.069290 1179 log.cc:1198] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:14.069676 1512 raft_consensus.cc:491] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.069834 1512 raft_consensus.cc:513] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.070655 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9cf200cbed2644eba78cf579cad06583" } new_table_name: "default.Table_65f2cee2d59e485fa3955cff5a8c4bff" modify_external_catalogs: true
I20250110 02:51:14.070722 1512 leader_election.cc:290] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.071288 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd85a070f74e9bb5dc4026935edcb5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:14.071413 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abd85a070f74e9bb5dc4026935edcb5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.071811 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_009562f3b1ed4d928d6fb21a604b051b" } modify_external_catalogs: true
W20250110 02:51:14.071846 709 leader_election.cc:343] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.071949 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_009562f3b1ed4d928d6fb21a604b051b" } modify_external_catalogs: true
W20250110 02:51:14.072011 711 leader_election.cc:343] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.072147 711 leader_election.cc:304] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.072501 1512 raft_consensus.cc:2743] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.073737 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f051d394f83d4eb39d7d6a458a478718" } modify_external_catalogs: true
I20250110 02:51:14.073907 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f051d394f83d4eb39d7d6a458a478718" } modify_external_catalogs: true
I20250110 02:51:14.074353 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0167bf0203804e32a12575dadcebdd98" } new_table_name: "default.Table_5bf91a5e0e004e1880fdf025d5d9ba38" modify_external_catalogs: true
I20250110 02:51:14.075448 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_77792a05daee44cfaaebbe693ef3cd65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.077735 1184 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.077915 1184 ts_tablet_manager.cc:1397] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.010s
I20250110 02:51:14.078438 1180 ts_tablet_manager.cc:1918] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:14.078572 1180 log.cc:1198] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2da6483de7f5475ea3d755a6efbda369
I20250110 02:51:14.078668 1179 ts_tablet_manager.cc:1939] T c65704c9d486444ebe8e4ffd4580f395 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.081084 1179 tablet_replica.cc:331] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.081447 1180 ts_tablet_manager.cc:1939] T 2da6483de7f5475ea3d755a6efbda369 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:14.081701 1179 raft_consensus.cc:2238] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.082041 1179 raft_consensus.cc:2267] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.082252 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 203) took 1872 ms. Trace:
I20250110 02:51:14.082262 881 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.082360 1180 rpcz_store.cc:276] 0110 02:51:12.209988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.211954 (+ 1966us) service_pool.cc:225] Handling call
0110 02:51:14.030586 (+1818632us) ts_tablet_manager.cc:1145] Deleting tablet 2da6483de7f5475ea3d755a6efbda369
0110 02:51:14.030593 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.058046 (+ 27453us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.070228 (+ 12182us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.082211 (+ 11983us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818279}
I20250110 02:51:14.082516 892 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:14.082629 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 32dd1e726f1a41d0894737b3ef3caf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32dd1e726f1a41d0894737b3ef3caf8f already in progress: creating tablet
W20250110 02:51:14.083006 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 already in progress: creating tablet
I20250110 02:51:14.083238 1179 ts_tablet_manager.cc:1905] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.083328 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:14.083436 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 60e534bf66f044b88d8cda0f3a1228f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e534bf66f044b88d8cda0f3a1228f4 already in progress: creating tablet
I20250110 02:51:14.083539 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:14.083914 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b375a932156d4abb908dbf327689b0f9 already in progress: creating tablet
W20250110 02:51:14.084259 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 514704cd935b44e8b760221224332509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514704cd935b44e8b760221224332509 already in progress: creating tablet
W20250110 02:51:14.084595 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 27a7885a80414f2193400672aa16a911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27a7885a80414f2193400672aa16a911 already in progress: creating tablet
W20250110 02:51:14.084934 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a81c031451104d5a93a1672b4b1918c5 already in progress: creating tablet
I20250110 02:51:14.085107 1184 raft_consensus.cc:357] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.085330 1184 raft_consensus.cc:383] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.085381 1178 ts_tablet_manager.cc:1918] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.085424 1184 raft_consensus.cc:738] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:14.085444 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e744cdc1fcb94d46b48b684763ecf1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e744cdc1fcb94d46b48b684763ecf1bf already in progress: creating tablet
I20250110 02:51:14.085646 1184 consensus_queue.cc:260] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.086540 1178 log.cc:1198] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/02d3f60c819f4824bc99d755e7442976
I20250110 02:51:14.086952 1178 ts_tablet_manager.cc:1939] T 02d3f60c819f4824bc99d755e7442976 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.087409 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 210) took 1795 ms. Trace:
I20250110 02:51:14.087518 1180 rpcz_store.cc:276] 0110 02:51:12.292019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.292181 (+ 162us) service_pool.cc:225] Handling call
0110 02:51:14.087278 (+1795097us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:14.087287 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.087375 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794804}
I20250110 02:51:14.088680 1184 ts_tablet_manager.cc:1428] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250110 02:51:14.088915 1184 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.089596 1180 tablet_replica.cc:331] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.090932 1178 tablet_replica.cc:331] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.091117 1178 raft_consensus.cc:2238] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.091282 1178 raft_consensus.cc:2267] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.091555 1180 raft_consensus.cc:2238] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.091709 1180 raft_consensus.cc:2267] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.092224 1180 ts_tablet_manager.cc:1905] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.094982 1184 tablet_bootstrap.cc:654] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.097546 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f15078fc22f4b32a48c7190b36e2192 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.099610 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.099895 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.100016 1386 raft_consensus.cc:491] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.100199 1386 raft_consensus.cc:513] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.100915 1386 leader_election.cc:290] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.101147 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:14.102396 996 leader_election.cc:343] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0b74812cb81c44fcb8786cc5dae328e3
I20250110 02:51:14.098934 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.099856 755 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.100474 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.101120 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.102218 745 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:14.104667 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77792a05daee44cfaaebbe693ef3cd65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.106753 1179 ts_tablet_manager.cc:1918] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.106931 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b74812cb81c44fcb8786cc5dae328e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.107156 917 raft_consensus.cc:2463] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.107607 994 leader_election.cc:304] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:14.108065 1386 raft_consensus.cc:2798] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:14.108155 1386 raft_consensus.cc:491] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:14.108264 1386 raft_consensus.cc:3054] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.108515 1179 log.cc:1198] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:14.102473 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2da6483de7f5475ea3d755a6efbda369 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
W20250110 02:51:14.109018 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2ef72e05f8ed4d46b759c7a04497653b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ef72e05f8ed4d46b759c7a04497653b already in progress: creating tablet
I20250110 02:51:14.109364 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 48cb20d47ace4fd49f9dc4e526cab702 (table default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]) successfully deleted
I20250110 02:51:14.109529 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c65704c9d486444ebe8e4ffd4580f395 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
W20250110 02:51:14.109709 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9cf980d9c4117b4ec3ca937cfcf01 already in progress: creating tablet
I20250110 02:51:14.109740 1179 ts_tablet_manager.cc:1939] T 2abd85a070f74e9bb5dc4026935edcb5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.110632 1184 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.110801 1184 ts_tablet_manager.cc:1397] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250110 02:51:14.110718 1386 raft_consensus.cc:513] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.111001 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2abd85a070f74e9bb5dc4026935edcb5 (table default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]) successfully deleted
I20250110 02:51:14.111662 1184 raft_consensus.cc:357] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.111927 1184 raft_consensus.cc:383] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.111995 1184 raft_consensus.cc:738] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.112239 1184 consensus_queue.cc:260] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.112686 1184 ts_tablet_manager.cc:1428] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250110 02:51:14.113191 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9cf980d9c4117b4ec3ca937cfcf01 already in progress: creating tablet
W20250110 02:51:14.113402 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6e4d3270e3a04cc99f90943b9573ec7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e4d3270e3a04cc99f90943b9573ec7e already in progress: creating tablet
W20250110 02:51:14.113619 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302af84fd8374ffa96a7503b54849536 already in progress: creating tablet
W20250110 02:51:14.113821 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4f1d916a5ea48708c2c1a61c92fe137 already in progress: creating tablet
W20250110 02:51:14.114024 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8c3ce95413df40369ee599176f91b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3ce95413df40369ee599176f91b892 already in progress: creating tablet
W20250110 02:51:14.114197 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d10ace37b96844328ccdb821dad3c5ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d10ace37b96844328ccdb821dad3c5ea already in progress: creating tablet
I20250110 02:51:14.114362 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 02d3f60c819f4824bc99d755e7442976 (table default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]) successfully deleted
W20250110 02:51:14.114573 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b43bb790e85343308a1bb1e1110bfb9c already in progress: creating tablet
W20250110 02:51:14.114724 996 leader_election.cc:343] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0b74812cb81c44fcb8786cc5dae328e3
W20250110 02:51:14.114904 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51adff7f50c4aec94e1892a50ec36c4 already in progress: creating tablet
I20250110 02:51:14.114959 1386 leader_election.cc:290] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.115238 1178 ts_tablet_manager.cc:1905] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:14.115536 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6e4d3270e3a04cc99f90943b9573ec7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e4d3270e3a04cc99f90943b9573ec7e already in progress: creating tablet
I20250110 02:51:14.115649 1180 ts_tablet_manager.cc:1918] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.115818 1180 log.cc:1198] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/02d3f60c819f4824bc99d755e7442976
I20250110 02:51:14.116209 1180 ts_tablet_manager.cc:1939] T 02d3f60c819f4824bc99d755e7442976 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:14.116343 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4f1d916a5ea48708c2c1a61c92fe137 already in progress: creating tablet
I20250110 02:51:14.117038 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 220) took 1697 ms. Trace:
I20250110 02:51:14.115576 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:14.117163 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8c3ce95413df40369ee599176f91b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3ce95413df40369ee599176f91b892 already in progress: creating tablet
W20250110 02:51:14.117909 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302af84fd8374ffa96a7503b54849536 already in progress: creating tablet
W20250110 02:51:14.118674 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51adff7f50c4aec94e1892a50ec36c4 already in progress: creating tablet
W20250110 02:51:14.119338 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b43bb790e85343308a1bb1e1110bfb9c already in progress: creating tablet
W20250110 02:51:14.119980 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d10ace37b96844328ccdb821dad3c5ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d10ace37b96844328ccdb821dad3c5ea already in progress: creating tablet
W20250110 02:51:14.120641 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 65ae1c19d2dc4778a1cebddbd505f907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ae1c19d2dc4778a1cebddbd505f907 already in progress: creating tablet
I20250110 02:51:14.121086 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b74812cb81c44fcb8786cc5dae328e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:14.121272 917 raft_consensus.cc:3054] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:14.121290 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f15078fc22f4b32a48c7190b36e2192 already in progress: creating tablet
W20250110 02:51:14.121909 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 46a4745356e3423f9d4892d242da1c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a4745356e3423f9d4892d242da1c48 already in progress: creating tablet
I20250110 02:51:14.122285 1179 tablet_replica.cc:331] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.122740 1179 raft_consensus.cc:2238] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.123564 917 raft_consensus.cc:2463] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:14.123595 1179 raft_consensus.cc:2267] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.124189 994 leader_election.cc:304] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:14.124593 1386 raft_consensus.cc:2798] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:14.124686 1179 ts_tablet_manager.cc:1905] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.124816 1386 raft_consensus.cc:695] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:14.125577 1184 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.125752 1386 consensus_queue.cc:237] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.127936 521 master_service.cc:946] ReplaceTablet: received request to replace tablet d98b789eaabe4191af40c9f713acb030 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.127957 1178 ts_tablet_manager.cc:1918] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.128062 1178 log.cc:1198] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/aa40d20288644b2a8d36ede967d16e79
I20250110 02:51:14.128392 1178 ts_tablet_manager.cc:1939] T aa40d20288644b2a8d36ede967d16e79 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.130512 1184 tablet_bootstrap.cc:654] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.130901 1178 tablet_replica.cc:331] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.131115 1178 raft_consensus.cc:2238] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.131276 1178 raft_consensus.cc:2267] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.117137 1180 rpcz_store.cc:276] 0110 02:51:12.419394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.419604 (+ 210us) service_pool.cc:225] Handling call
0110 02:51:14.089539 (+1669935us) ts_tablet_manager.cc:1145] Deleting tablet 02d3f60c819f4824bc99d755e7442976
0110 02:51:14.089549 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.107541 (+ 17992us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.115540 (+ 7999us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.116986 (+ 1446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1633246}
I20250110 02:51:14.131944 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 221) took 1660 ms. Trace:
I20250110 02:51:14.132488 1178 ts_tablet_manager.cc:1905] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.132668 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 02d3f60c819f4824bc99d755e7442976 (table default.Table_009422f290cd4fc9983b13a098c7b13b [id=78ead7d609e14db8af73d5ca0d11e8a7]) successfully deleted
I20250110 02:51:14.132910 1180 rpcz_store.cc:276] 0110 02:51:12.471836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.471895 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:14.131761 (+1659866us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:14.131774 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.131881 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659660}
I20250110 02:51:14.134048 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 224) took 1661 ms. Trace:
I20250110 02:51:14.135704 1180 rpcz_store.cc:276] 0110 02:51:12.472146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.472196 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:14.133897 (+1661701us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:14.133908 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.133999 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660163}
I20250110 02:51:14.136423 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 222) took 1664 ms. Trace:
I20250110 02:51:14.137002 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:14.137392 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3a9eaf548274b73a5652ba314dab70c consensus state has no leader
W20250110 02:51:14.137944 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d970e2fe72b4727968ca605b3e772cc consensus state has no leader
W20250110 02:51:14.138417 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1edb4e06e40c43438882e1eb622d788e consensus state has no leader
I20250110 02:51:14.138443 1180 rpcz_store.cc:276] 0110 02:51:12.471957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.473084 (+ 1127us) service_pool.cc:225] Handling call
0110 02:51:14.136271 (+1663187us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:14.136282 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.136372 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":1662525}
I20250110 02:51:14.139513 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 223) took 1667 ms. Trace:
I20250110 02:51:14.140085 1180 rpcz_store.cc:276] 0110 02:51:12.472050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.473344 (+ 1294us) service_pool.cc:225] Handling call
0110 02:51:14.139363 (+1666019us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:14.139372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.139461 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664239}
I20250110 02:51:14.141026 1184 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.141103 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6bf83ecf29b84c6babc878760e92fce1" } new_table_name: "default.Table_796db82db8ec4b7ebb6d3d3ab676114f" modify_external_catalogs: true
I20250110 02:51:14.141481 1180 tablet_replica.cc:331] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.141690 1180 raft_consensus.cc:2238] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:14.142127 1180 pending_rounds.cc:62] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:14.142408 1184 ts_tablet_manager.cc:1397] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20250110 02:51:14.142675 1180 pending_rounds.cc:69] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611732685803520 op_type: NO_OP noop_request { }
W20250110 02:51:14.143713 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5e183122d13a4c38988bf2fe6f7a3694 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e183122d13a4c38988bf2fe6f7a3694 already in progress: creating tablet
W20250110 02:51:14.143939 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet dad0d561a8ef498ba13bc813d4cd82fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dad0d561a8ef498ba13bc813d4cd82fa already in progress: creating tablet
I20250110 02:51:14.144110 1180 raft_consensus.cc:2883] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250110 02:51:14.144161 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7d4422eca5604d149e73c2a669d07af0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d4422eca5604d149e73c2a669d07af0 already in progress: creating tablet
I20250110 02:51:14.144335 1180 raft_consensus.cc:2267] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:14.144351 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fc9826ef554a5699e365d10d02e97a already in progress: creating tablet
I20250110 02:51:14.145318 1180 ts_tablet_manager.cc:1905] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.143731 1184 raft_consensus.cc:357] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.148586 1184 raft_consensus.cc:383] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.148926 1184 raft_consensus.cc:738] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.149144 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.149922 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.150126 1184 consensus_queue.cc:260] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.146078 1179 ts_tablet_manager.cc:1918] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.150765 1179 log.cc:1198] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/89f6797b1076468a93fb0b510c553bed
I20250110 02:51:14.151124 1179 ts_tablet_manager.cc:1939] T 89f6797b1076468a93fb0b510c553bed P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.153091 1179 tablet_replica.cc:331] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.153342 1179 raft_consensus.cc:2238] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.153510 1179 raft_consensus.cc:2267] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.154961 1179 ts_tablet_manager.cc:1905] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.161834 1179 ts_tablet_manager.cc:1918] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.162822 1178 ts_tablet_manager.cc:1918] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.163023 1178 log.cc:1198] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/4e2c01af512f460daa98b19444f72172
I20250110 02:51:14.163441 1178 ts_tablet_manager.cc:1939] T 4e2c01af512f460daa98b19444f72172 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.163360 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc32061409246a2b280dde4f9180a8e (DEFAULT_TABLE table=default.Table_73259c80bde24fd3a2440bdce73bd88e [id=0afe16dcfb0c4ef0897ceb5fb2993d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.163800 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc32061409246a2b280dde4f9180a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.164777 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4 (DEFAULT_TABLE table=default.Table_8cf4c26686b84eb59c11d17010f8a679 [id=d37919aec6fc44ba9c1800b84d5fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.165198 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.167255 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 7564208da99449cd97338a41b263e750 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.167433 1178 tablet_replica.cc:331] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.167627 1178 raft_consensus.cc:2238] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.168020 1178 raft_consensus.cc:2267] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.168239 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564208da99449cd97338a41b263e750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.170784 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (DEFAULT_TABLE table=default.Table_105ea910a3c447a6abb0c0919ab8ee06 [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.171147 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0b2ec1d5814144bc58261e6c6e8d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.173307 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1d2e132b16324d3ea79f29a7339f4d5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.173791 1386 raft_consensus.cc:491] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:14.173959 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d2e132b16324d3ea79f29a7339f4d5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.173934 1386 raft_consensus.cc:513] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.174558 1386 leader_election.cc:290] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.175230 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c99830786e0e4b7b956e604dd3a5fcc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.175616 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:14.175719 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c99830786e0e4b7b956e604dd3a5fcc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:14.176383 994 leader_election.cc:343] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.177310 1178 ts_tablet_manager.cc:1905] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:14.177166 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 54225.2us
I20250110 02:51:14.178769 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet aa40d20288644b2a8d36ede967d16e79 (table default.Table_8df25635dd7e4d58818dc14fc678e221 [id=4893b36e085445c4a454cf47ebe1955b]) successfully deleted
W20250110 02:51:14.179380 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 46a4745356e3423f9d4892d242da1c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a4745356e3423f9d4892d242da1c48 already in progress: creating tablet
W20250110 02:51:14.179854 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f15078fc22f4b32a48c7190b36e2192 already in progress: creating tablet
W20250110 02:51:14.180197 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 65ae1c19d2dc4778a1cebddbd505f907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ae1c19d2dc4778a1cebddbd505f907 already in progress: creating tablet
I20250110 02:51:14.180615 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 4e2c01af512f460daa98b19444f72172 (table default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]) successfully deleted
W20250110 02:51:14.181074 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2e10e7654cec495bb532fb00e73c020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e10e7654cec495bb532fb00e73c020d already in progress: creating tablet
W20250110 02:51:14.181638 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e25f9643ac0476a90c93fb9aeb921a3 already in progress: creating tablet
I20250110 02:51:14.161931 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.162600 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (DEFAULT_TABLE table=default.Table_105ea910a3c447a6abb0c0919ab8ee06 [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.182857 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0b2ec1d5814144bc58261e6c6e8d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.183053 745 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4 (DEFAULT_TABLE table=default.Table_8cf4c26686b84eb59c11d17010f8a679 [id=d37919aec6fc44ba9c1800b84d5fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.183444 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.183624 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (DEFAULT_TABLE table=default.Table_105ea910a3c447a6abb0c0919ab8ee06 [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.183957 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0b2ec1d5814144bc58261e6c6e8d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.184693 1386 raft_consensus.cc:491] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.184826 1386 raft_consensus.cc:513] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.185379 1386 leader_election.cc:290] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.186630 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc32061409246a2b280dde4f9180a8e (DEFAULT_TABLE table=default.Table_73259c80bde24fd3a2440bdce73bd88e [id=0afe16dcfb0c4ef0897ceb5fb2993d7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:14.186663 994 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 0c2412420dc54258a6921ead194c1587
I20250110 02:51:14.164044 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7564208da99449cd97338a41b263e750 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.166769 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc32061409246a2b280dde4f9180a8e (DEFAULT_TABLE table=default.Table_73259c80bde24fd3a2440bdce73bd88e [id=0afe16dcfb0c4ef0897ceb5fb2993d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.187000 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc32061409246a2b280dde4f9180a8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.187036 996 leader_election.cc:343] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0c2412420dc54258a6921ead194c1587
I20250110 02:51:14.187160 996 leader_election.cc:304] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:14.175045 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:14.187476 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564208da99449cd97338a41b263e750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.187595 776 raft_consensus.cc:2463] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.187688 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc32061409246a2b280dde4f9180a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.187916 1507 raft_consensus.cc:2743] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.188083 1507 raft_consensus.cc:491] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.188191 1507 raft_consensus.cc:513] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.188760 1507 leader_election.cc:290] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.189422 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.189603 917 raft_consensus.cc:2463] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.189971 994 leader_election.cc:304] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:14.190317 996 leader_election.cc:304] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:14.190660 1467 raft_consensus.cc:2798] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:14.190636 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7564208da99449cd97338a41b263e750 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.190752 1467 raft_consensus.cc:491] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:14.190811 1467 raft_consensus.cc:3054] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.190961 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7564208da99449cd97338a41b263e750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.191126 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a3ba2bc51a3b4fc6b6451349e08b106d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.191613 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3ba2bc51a3b4fc6b6451349e08b106d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.193073 1467 raft_consensus.cc:513] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.193702 1467 leader_election.cc:290] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.193852 1467 raft_consensus.cc:2798] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:14.193917 1467 raft_consensus.cc:491] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:14.194003 1467 raft_consensus.cc:3054] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.194640 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4 (DEFAULT_TABLE table=default.Table_8cf4c26686b84eb59c11d17010f8a679 [id=d37919aec6fc44ba9c1800b84d5fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.194967 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.195261 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:14.195427 917 raft_consensus.cc:3054] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.195981 1037 tablet_service.cc:1467] Processing CreateTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 (DEFAULT_TABLE table=default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.196504 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015a41a03f84c91a45e46e4b7b5e7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.198194 917 raft_consensus.cc:2463] T 8162ceb17c714614b34beb33a7633ff0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:14.198629 994 leader_election.cc:304] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:14.198652 894 tablet_service.cc:1467] Processing CreateTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 (DEFAULT_TABLE table=default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.198910 1386 raft_consensus.cc:2798] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:14.198964 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015a41a03f84c91a45e46e4b7b5e7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.199019 1386 raft_consensus.cc:695] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:14.199309 1386 consensus_queue.cc:237] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.202615 882 tablet_service.cc:1467] Processing CreateTablet for tablet c00e97e496b54dcf82b7182591d6f4aa (DEFAULT_TABLE table=default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.202951 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00e97e496b54dcf82b7182591d6f4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.204730 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:14.177096 1184 ts_tablet_manager.cc:1428] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20250110 02:51:14.205437 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c00e97e496b54dcf82b7182591d6f4aa (DEFAULT_TABLE table=default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.205817 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00e97e496b54dcf82b7182591d6f4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.206012 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 89f6797b1076468a93fb0b510c553bed (table default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]) successfully deleted
W20250110 02:51:14.206427 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2e10e7654cec495bb532fb00e73c020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e10e7654cec495bb532fb00e73c020d already in progress: creating tablet
I20250110 02:51:14.206625 887 tablet_service.cc:1467] Processing CreateTablet for tablet bfd120d8edf14151b7ebed3b74e2b399 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.206960 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd120d8edf14151b7ebed3b74e2b399. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.206914 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e25f9643ac0476a90c93fb9aeb921a3 already in progress: creating tablet
W20250110 02:51:14.207316 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f6df074974974d81a83420e7a3a78070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df074974974d81a83420e7a3a78070 already in progress: creating tablet
I20250110 02:51:14.208752 1023 tablet_service.cc:1467] Processing CreateTablet for tablet bfd120d8edf14151b7ebed3b74e2b399 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.209340 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd120d8edf14151b7ebed3b74e2b399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.210235 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8162ceb17c714614b34beb33a7633ff0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:14.210235 776 raft_consensus.cc:2463] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.212833 754 tablet_service.cc:1467] Processing CreateTablet for tablet c00e97e496b54dcf82b7182591d6f4aa (DEFAULT_TABLE table=default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.214005 749 tablet_service.cc:1467] Processing CreateTablet for tablet bfd120d8edf14151b7ebed3b74e2b399 (DEFAULT_TABLE table=default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:14.215857 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb726d2c9a04963b92a3fb625cf84f3 already in progress: creating tablet
I20250110 02:51:14.216413 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00e97e496b54dcf82b7182591d6f4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.217653 1467 raft_consensus.cc:513] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.218322 1467 leader_election.cc:290] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:14.217295 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b382d634db9642df8e00c5ccaa5787a9 already in progress: creating tablet
I20250110 02:51:14.219655 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:14.219672 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f56778a0244378ad8499a4ceef3510" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:14.220003 776 raft_consensus.cc:3054] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.221120 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd120d8edf14151b7ebed3b74e2b399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.221334 748 tablet_service.cc:1467] Processing CreateTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 (DEFAULT_TABLE table=default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:14.221661 994 leader_election.cc:343] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.223942 776 raft_consensus.cc:2463] T d5f56778a0244378ad8499a4ceef3510 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:14.226424 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e79175853474cbea951f5a5686bf630 already in progress: creating tablet
I20250110 02:51:14.227373 996 leader_election.cc:304] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:14.227960 1467 raft_consensus.cc:2798] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:14.228155 1467 raft_consensus.cc:695] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:14.228452 1467 consensus_queue.cc:237] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.234449 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e015a41a03f84c91a45e46e4b7b5e7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.244449 1178 ts_tablet_manager.cc:1918] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.244652 1178 log.cc:1198] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/0b74812cb81c44fcb8786cc5dae328e3
I20250110 02:51:14.245057 1178 ts_tablet_manager.cc:1939] T 0b74812cb81c44fcb8786cc5dae328e3 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.246742 1179 log.cc:1198] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:14.246650 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6f9765add6c947e8b8154e36e6d358dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.247519 1179 ts_tablet_manager.cc:1939] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.249014 1181 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.252025 1181 ts_tablet_manager.cc:1397] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.254s user 0.021s sys 0.009s
W20250110 02:51:14.256999 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f9765add6c947e8b8154e36e6d358dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.251008 1184 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.261176 1184 tablet_bootstrap.cc:654] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.255283 1181 raft_consensus.cc:357] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.270977 1179 tablet_replica.cc:331] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.271286 1179 raft_consensus.cc:2238] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.271518 1179 raft_consensus.cc:2267] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.272259 1179 ts_tablet_manager.cc:1905] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.277179 1184 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.277804 522 catalog_manager.cc:5526] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:14.280584 1180 ts_tablet_manager.cc:1918] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:14.279701 776 raft_consensus.cc:1270] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:14.281385 1184 ts_tablet_manager.cc:1397] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.003s
I20250110 02:51:14.281751 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d98b789eaabe4191af40c9f713acb030
I20250110 02:51:14.282099 1367 consensus_queue.cc:1035] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:14.282316 521 catalog_manager.cc:6596] ReplaceTablet: tablet d98b789eaabe4191af40c9f713acb030 deleted and replaced by tablet 34acf24a32a54be58d746e86c5be9cfe
I20250110 02:51:14.284363 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_78b575ae2603466d9703e8780d3cd413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.284823 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b575ae2603466d9703e8780d3cd413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.292263 1184 raft_consensus.cc:357] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.292541 1184 raft_consensus.cc:383] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.292596 1184 raft_consensus.cc:738] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.292840 1184 consensus_queue.cc:260] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.293501 1184 ts_tablet_manager.cc:1428] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20250110 02:51:14.293853 1184 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.281620 1059 raft_consensus.cc:1270] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:14.294978 1180 log.cc:1198] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/aa40d20288644b2a8d36ede967d16e79
I20250110 02:51:14.268342 1181 raft_consensus.cc:383] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.295388 1181 raft_consensus.cc:738] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.295439 1180 ts_tablet_manager.cc:1939] T aa40d20288644b2a8d36ede967d16e79 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:14.295621 1181 consensus_queue.cc:260] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.295727 1367 consensus_queue.cc:1035] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:14.296872 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 0b74812cb81c44fcb8786cc5dae328e3 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]) successfully deleted
I20250110 02:51:14.296986 1181 ts_tablet_manager.cc:1428] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.044s user 0.003s sys 0.000s
I20250110 02:51:14.297376 1181 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.298918 1188 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.299100 1188 ts_tablet_manager.cc:1397] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.265s user 0.005s sys 0.001s
I20250110 02:51:14.299670 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 227) took 1571 ms. Trace:
I20250110 02:51:14.299778 1180 rpcz_store.cc:276] 0110 02:51:12.727834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.727970 (+ 136us) service_pool.cc:225] Handling call
0110 02:51:14.141403 (+1413433us) ts_tablet_manager.cc:1145] Deleting tablet aa40d20288644b2a8d36ede967d16e79
0110 02:51:14.141412 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.180321 (+ 38909us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.280443 (+100122us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:14.299618 (+ 19175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406943}
I20250110 02:51:14.300081 1188 raft_consensus.cc:357] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.300169 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 226) took 1573 ms. Trace:
I20250110 02:51:14.300329 1188 raft_consensus.cc:383] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.300262 1180 rpcz_store.cc:276] 0110 02:51:12.727040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.727127 (+ 87us) service_pool.cc:225] Handling call
0110 02:51:14.300028 (+1572901us) ts_tablet_manager.cc:1145] Deleting tablet 1602ade49b7442d6b8e6f4b85d2e80a1
0110 02:51:14.300036 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.300122 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5153,"tablet-delete.queue_time_us":1565498}
I20250110 02:51:14.300516 1188 raft_consensus.cc:738] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.300794 1188 consensus_queue.cc:260] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.302171 1188 ts_tablet_manager.cc:1428] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:14.302546 1188 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:14.304852 1188 tablet_bootstrap.cc:654] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:14.305819 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f6df074974974d81a83420e7a3a78070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df074974974d81a83420e7a3a78070 already in progress: creating tablet
W20250110 02:51:14.306113 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb726d2c9a04963b92a3fb625cf84f3 already in progress: creating tablet
W20250110 02:51:14.306389 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b382d634db9642df8e00c5ccaa5787a9 already in progress: creating tablet
W20250110 02:51:14.306633 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e79175853474cbea951f5a5686bf630 already in progress: creating tablet
I20250110 02:51:14.307215 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d98b789eaabe4191af40c9f713acb030 with delete_type TABLET_DATA_DELETED (replaced by tablet 34acf24a32a54be58d746e86c5be9cfe) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.307472 1178 tablet_replica.cc:331] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.312659 1181 tablet_bootstrap.cc:654] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.313462 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 229) took 1560 ms. Trace:
I20250110 02:51:14.314620 1180 rpcz_store.cc:276] 0110 02:51:12.740185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.740245 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:14.300493 (+1560248us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:14.300499 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.300574 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555769}
I20250110 02:51:14.315752 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 228) took 1587 ms. Trace:
I20250110 02:51:14.315938 1180 rpcz_store.cc:276] 0110 02:51:12.728765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.728842 (+ 77us) service_pool.cc:225] Handling call
0110 02:51:14.315590 (+1586748us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:14.315597 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.315708 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564457}
I20250110 02:51:14.316454 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 230) took 1571 ms. Trace:
I20250110 02:51:14.317107 1180 rpcz_store.cc:276] 0110 02:51:12.745229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.745831 (+ 602us) service_pool.cc:225] Handling call
0110 02:51:14.316321 (+1570490us) ts_tablet_manager.cc:1145] Deleting tablet 344184e1c98d422095cb254f5b2bc792
0110 02:51:14.316330 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.316411 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564891}
I20250110 02:51:14.317445 1184 tablet_bootstrap.cc:654] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.320505 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 232) took 1563 ms. Trace:
I20250110 02:51:14.320660 1180 rpcz_store.cc:276] 0110 02:51:12.756803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.756851 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:14.320373 (+1563522us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:14.320383 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.320461 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563339}
I20250110 02:51:14.321239 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 231) took 1575 ms. Trace:
I20250110 02:51:14.321377 1180 rpcz_store.cc:276] 0110 02:51:12.745329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.745857 (+ 528us) service_pool.cc:225] Handling call
0110 02:51:14.321101 (+1575244us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:14.321111 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.321191 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563329}
I20250110 02:51:14.322290 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 242) took 1497 ms. Trace:
I20250110 02:51:14.323105 1180 rpcz_store.cc:276] 0110 02:51:12.824414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.825131 (+ 717us) service_pool.cc:225] Handling call
0110 02:51:14.322138 (+1497007us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:14.322148 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.322236 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491853}
I20250110 02:51:14.326190 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 243) took 1501 ms. Trace:
I20250110 02:51:14.327100 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d98b789eaabe4191af40c9f713acb030 with delete_type TABLET_DATA_DELETED (replaced by tablet 34acf24a32a54be58d746e86c5be9cfe) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.327723 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (table default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]) successfully deleted
I20250110 02:51:14.327796 1537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:39025 every 8 connection(s)
I20250110 02:51:14.328903 1179 ts_tablet_manager.cc:1918] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.329102 1179 log.cc:1198] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8940c201aa104b0b8bc8d866859fe8ab
W20250110 02:51:14.330770 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75ca5369caf3415ea508352362409656 consensus state has no leader
I20250110 02:51:14.326841 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d98b789eaabe4191af40c9f713acb030 with delete_type TABLET_DATA_DELETED (replaced by tablet 34acf24a32a54be58d746e86c5be9cfe) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.328593 1181 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.331207 1241 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:39025
I20250110 02:51:14.331342 1181 ts_tablet_manager.cc:1397] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20250110 02:51:14.332038 1181 raft_consensus.cc:357] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.332231 1181 raft_consensus.cc:383] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.332291 1181 raft_consensus.cc:738] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.332530 1181 consensus_queue.cc:260] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.333081 1181 ts_tablet_manager.cc:1428] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:14.333123 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-2/data/info.pb
I20250110 02:51:14.333364 1181 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.334012 1546 tablet.cc:1721] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:14.334617 1178 raft_consensus.cc:2238] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:14.334919 1181 tablet_bootstrap.cc:654] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.335058 1178 raft_consensus.cc:2267] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.336184 1180 rpcz_store.cc:276] 0110 02:51:12.824528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.825156 (+ 628us) service_pool.cc:225] Handling call
0110 02:51:14.325460 (+1500304us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:14.325469 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.326145 (+ 676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492870}
I20250110 02:51:14.336827 1179 ts_tablet_manager.cc:1939] T 8940c201aa104b0b8bc8d866859fe8ab P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.337781 1179 tablet_replica.cc:331] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.337836 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 1241
I20250110 02:51:14.338003 1179 raft_consensus.cc:2238] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.338161 1179 raft_consensus.cc:2267] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.338940 1184 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.339052 1184 ts_tablet_manager.cc:1397] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250110 02:51:14.340236 1184 raft_consensus.cc:357] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.340463 1184 raft_consensus.cc:383] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.340543 1184 raft_consensus.cc:738] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.340795 1184 consensus_queue.cc:260] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.341398 1184 ts_tablet_manager.cc:1428] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:14.341631 1184 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.342677 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 255) took 1410 ms. Trace:
I20250110 02:51:14.342820 1180 rpcz_store.cc:276] 0110 02:51:12.932502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.932544 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:14.342481 (+1409937us) ts_tablet_manager.cc:1145] Deleting tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8
0110 02:51:14.342492 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.342627 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409753}
I20250110 02:51:14.343436 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 258) took 1370 ms. Trace:
I20250110 02:51:14.343561 1180 rpcz_store.cc:276] 0110 02:51:12.977110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.982122 (+ 5012us) service_pool.cc:225] Handling call
0110 02:51:14.343275 (+1361153us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:14.343285 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.343381 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346464}
I20250110 02:51:14.334136 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_26e7f79065c5436eaf670e8a52c72baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.344102 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 250) took 1413 ms. Trace:
I20250110 02:51:14.334666 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 24c4042c38de4969889f4821176fc7bb
I20250110 02:51:14.344190 1180 rpcz_store.cc:276] 0110 02:51:12.930847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.930911 (+ 64us) service_pool.cc:225] Handling call
0110 02:51:14.343900 (+1412989us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:14.343911 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.344056 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341752}
I20250110 02:51:14.344588 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 252) took 1413 ms. Trace:
I20250110 02:51:14.344668 1180 rpcz_store.cc:276] 0110 02:51:12.931114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.931588 (+ 474us) service_pool.cc:225] Handling call
0110 02:51:14.344464 (+1412876us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:14.344471 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.344544 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340922}
I20250110 02:51:14.345072 1178 ts_tablet_manager.cc:1905] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.345175 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 259) took 1365 ms. Trace:
I20250110 02:51:14.345242 1180 rpcz_store.cc:276] 0110 02:51:12.979588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:12.998022 (+ 18434us) service_pool.cc:225] Handling call
0110 02:51:14.344922 (+1346900us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:14.344939 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.345130 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1330593}
I20250110 02:51:14.345642 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 271) took 1114 ms. Trace:
I20250110 02:51:14.345737 1180 rpcz_store.cc:276] 0110 02:51:13.231607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.232887 (+ 1280us) service_pool.cc:225] Handling call
0110 02:51:14.345520 (+1112633us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:14.345527 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.345598 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112415}
I20250110 02:51:14.345870 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cebc63fd7b144a688a2c94924bbf5b08" } new_table_name: "default.Table_7cd682c4f72745b5aa221478659d3350" modify_external_catalogs: true
I20250110 02:51:14.346122 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 276) took 1114 ms. Trace:
I20250110 02:51:14.346191 1180 rpcz_store.cc:276] 0110 02:51:13.232070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.233129 (+ 1059us) service_pool.cc:225] Handling call
0110 02:51:14.346007 (+1112878us) ts_tablet_manager.cc:1145] Deleting tablet 514704cd935b44e8b760221224332509
0110 02:51:14.346015 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.346082 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112752}
I20250110 02:51:14.346597 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 277) took 1114 ms. Trace:
I20250110 02:51:14.346669 1180 rpcz_store.cc:276] 0110 02:51:13.232164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.233266 (+ 1102us) service_pool.cc:225] Handling call
0110 02:51:14.346479 (+1113213us) ts_tablet_manager.cc:1145] Deleting tablet e744cdc1fcb94d46b48b684763ecf1bf
0110 02:51:14.346487 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.346554 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113078}
I20250110 02:51:14.335582 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:14.347388 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 278) took 1115 ms. Trace:
I20250110 02:51:14.347458 1180 rpcz_store.cc:276] 0110 02:51:13.232232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.233411 (+ 1179us) service_pool.cc:225] Handling call
0110 02:51:14.347260 (+1113849us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:14.347267 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.347345 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113736}
I20250110 02:51:14.347874 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 274) took 1115 ms. Trace:
I20250110 02:51:14.347857 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8f8cd5df24f147c5ae832efed9aad406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.347963 1180 rpcz_store.cc:276] 0110 02:51:13.231897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.233914 (+ 2017us) service_pool.cc:225] Handling call
0110 02:51:14.347735 (+1113821us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:14.347745 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.347826 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113663}
W20250110 02:51:14.348397 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f8cd5df24f147c5ae832efed9aad406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.348884 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 275) took 1116 ms. Trace:
I20250110 02:51:14.348969 1180 rpcz_store.cc:276] 0110 02:51:13.231972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.234469 (+ 2497us) service_pool.cc:225] Handling call
0110 02:51:14.348230 (+1113761us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:14.348238 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.348844 (+ 606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113609}
I20250110 02:51:14.336123 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c36e12f8b3e64b41980128408d36846f" } modify_external_catalogs: true
I20250110 02:51:14.350081 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c36e12f8b3e64b41980128408d36846f" } modify_external_catalogs: true
I20250110 02:51:14.350672 1178 ts_tablet_manager.cc:1918] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:14.350796 1178 log.cc:1198] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d98b789eaabe4191af40c9f713acb030
I20250110 02:51:14.350895 1179 ts_tablet_manager.cc:1905] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.351246 1178 ts_tablet_manager.cc:1939] T d98b789eaabe4191af40c9f713acb030 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:14.353861 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e7f79065c5436eaf670e8a52c72baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.368240 1556 sys_catalog.cc:263] Verifying existing consensus state
W20250110 02:51:14.369189 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcc9a4a842914a6f878d01194cd9a5f7 already in progress: creating tablet
I20250110 02:51:14.370541 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.371413 1178 tablet_replica.cc:331] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.371687 1178 raft_consensus.cc:2238] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.371858 1178 raft_consensus.cc:2267] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.372463 1178 ts_tablet_manager.cc:1905] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.372475 1554 tablet.cc:1721] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:14.375855 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Bootstrap starting.
I20250110 02:51:14.378964 1178 ts_tablet_manager.cc:1918] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.379148 1178 log.cc:1198] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/64c77ec8c7724d5fb408a8bfa53811eb
I20250110 02:51:14.379570 1178 ts_tablet_manager.cc:1939] T 64c77ec8c7724d5fb408a8bfa53811eb P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.381089 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.382046 881 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.382476 1181 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.382625 1181 ts_tablet_manager.cc:1397] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
W20250110 02:51:14.383683 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d20177229f7b4f11875e0007afb1dd00 already in progress: creating tablet
I20250110 02:51:14.383714 1181 raft_consensus.cc:357] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.383921 1181 raft_consensus.cc:383] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.384021 1181 raft_consensus.cc:738] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.384246 1181 consensus_queue.cc:260] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.384683 1181 ts_tablet_manager.cc:1428] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:14.384979 1181 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.383240 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]) successfully deleted
I20250110 02:51:14.385238 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 64c77ec8c7724d5fb408a8bfa53811eb (table default.Table_e676f1148b5f4988a2be33981eec57c3 [id=d7bbf5b39b224708974b5f6da6429fe7]) successfully deleted
I20250110 02:51:14.388190 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 270) took 1155 ms. Trace:
I20250110 02:51:14.388327 1180 rpcz_store.cc:276] 0110 02:51:13.231485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.268418 (+ 36933us) service_pool.cc:225] Handling call
0110 02:51:14.379214 (+1110796us) ts_tablet_manager.cc:1145] Deleting tablet 60e534bf66f044b88d8cda0f3a1228f4
0110 02:51:14.379224 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.387341 (+ 8117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1086952}
I20250110 02:51:14.388973 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 287) took 1067 ms. Trace:
I20250110 02:51:14.389103 1180 rpcz_store.cc:276] 0110 02:51:13.321717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:13.322231 (+ 514us) service_pool.cc:225] Handling call
0110 02:51:14.388752 (+1066521us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:14.388762 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:14.388840 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064687}
I20250110 02:51:14.390573 1188 tablet_bootstrap.cc:492] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.390749 1188 ts_tablet_manager.cc:1397] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20250110 02:51:14.391119 1184 tablet_bootstrap.cc:654] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.391528 1188 raft_consensus.cc:357] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.391760 1188 raft_consensus.cc:383] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.391837 1188 raft_consensus.cc:738] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:14.392254 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4042c38de4969889f4821176fc7bb already in progress: creating tablet
W20250110 02:51:14.385533 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 344184e1c98d422095cb254f5b2bc792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344184e1c98d422095cb254f5b2bc792 already in progress: creating tablet
W20250110 02:51:14.392838 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f5e88d1bbc304b4ebbc849a7d783052e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e88d1bbc304b4ebbc849a7d783052e already in progress: creating tablet
I20250110 02:51:14.393344 1181 tablet_bootstrap.cc:654] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.393611 1556 log.cc:826] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:14.393905 1180 tablet_replica.cc:331] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.394531 1180 raft_consensus.cc:2238] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.395680 1180 raft_consensus.cc:2267] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:14.395790 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00e97e496b54dcf82b7182591d6f4aa already in progress: creating tablet
W20250110 02:51:14.396037 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f4778347f054b86af697dbe3b287263 already in progress: creating tablet
I20250110 02:51:14.396561 1180 ts_tablet_manager.cc:1905] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:14.396849 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b6d57f3e9a44d178300d3505d504854 already in progress: creating tablet
W20250110 02:51:14.397061 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d5ddbc6b02742609bc2ca84c863a9d0 already in progress: creating tablet
W20250110 02:51:14.397248 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 already in progress: creating tablet
W20250110 02:51:14.397446 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 0127de45264f4660b843734a530eb310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0127de45264f4660b843734a530eb310 already in progress: creating tablet
W20250110 02:51:14.397624 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f66603a0ee8c4082807c5a6aabe7e339 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f66603a0ee8c4082807c5a6aabe7e339 already in progress: creating tablet
I20250110 02:51:14.397606 1549 tablet.cc:1721] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:14.399575 1179 ts_tablet_manager.cc:1918] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.404250 1179 log.cc:1198] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/344184e1c98d422095cb254f5b2bc792
I20250110 02:51:14.404961 1179 ts_tablet_manager.cc:1939] T 344184e1c98d422095cb254f5b2bc792 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.407547 1188 consensus_queue.cc:260] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:14.408074 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff9b10c419242cf85ef1b708bd5a862 already in progress: creating tablet
W20250110 02:51:14.408715 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 32dd1e726f1a41d0894737b3ef3caf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32dd1e726f1a41d0894737b3ef3caf8f already in progress: creating tablet
I20250110 02:51:14.408824 1188 ts_tablet_manager.cc:1428] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
W20250110 02:51:14.409047 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 already in progress: creating tablet
I20250110 02:51:14.409363 1188 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:14.409569 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 514704cd935b44e8b760221224332509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514704cd935b44e8b760221224332509 already in progress: creating tablet
I20250110 02:51:14.411161 1188 tablet_bootstrap.cc:654] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:14.411384 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e744cdc1fcb94d46b48b684763ecf1bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e744cdc1fcb94d46b48b684763ecf1bf already in progress: creating tablet
W20250110 02:51:14.411598 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a81c031451104d5a93a1672b4b1918c5 already in progress: creating tablet
W20250110 02:51:14.411871 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 27a7885a80414f2193400672aa16a911 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27a7885a80414f2193400672aa16a911 already in progress: creating tablet
W20250110 02:51:14.412113 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b375a932156d4abb908dbf327689b0f9 already in progress: creating tablet
I20250110 02:51:14.412251 1179 tablet_replica.cc:331] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.414600 1179 raft_consensus.cc:2238] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.415793 1179 raft_consensus.cc:2267] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.415630 1184 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.416307 1184 ts_tablet_manager.cc:1397] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.000s
I20250110 02:51:14.419795 1467 raft_consensus.cc:491] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.419975 1467 raft_consensus.cc:513] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.420624 1467 leader_election.cc:290] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:14.424188 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 60e534bf66f044b88d8cda0f3a1228f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e534bf66f044b88d8cda0f3a1228f4 already in progress: creating tablet
W20250110 02:51:14.424401 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9cf980d9c4117b4ec3ca937cfcf01 already in progress: creating tablet
W20250110 02:51:14.423902 510 catalog_manager.cc:4945] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): alter failed for tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]),no further retry: Not found: Tablet not found: d98b789eaabe4191af40c9f713acb030
I20250110 02:51:14.421003 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:14.425673 994 leader_election.cc:343] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.426268 1181 tablet_bootstrap.cc:492] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.426447 1181 ts_tablet_manager.cc:1397] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.002s
W20250110 02:51:14.424610 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8c3ce95413df40369ee599176f91b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3ce95413df40369ee599176f91b892 already in progress: creating tablet
I20250110 02:51:14.427213 1181 raft_consensus.cc:357] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.427412 1181 raft_consensus.cc:383] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.427486 1181 raft_consensus.cc:738] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.427733 1181 consensus_queue.cc:260] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.428668 1181 ts_tablet_manager.cc:1428] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250110 02:51:14.428682 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4f1d916a5ea48708c2c1a61c92fe137 already in progress: creating tablet
I20250110 02:51:14.430634 1181 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.432091 1181 tablet_bootstrap.cc:654] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.422577 1180 ts_tablet_manager.cc:1918] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.432906 1180 log.cc:1198] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/0c2412420dc54258a6921ead194c1587
I20250110 02:51:14.433248 1180 ts_tablet_manager.cc:1939] T 0c2412420dc54258a6921ead194c1587 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:14.433317 963 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:14.435094 1106 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
W20250110 02:51:14.437768 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6e4d3270e3a04cc99f90943b9573ec7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e4d3270e3a04cc99f90943b9573ec7e already in progress: creating tablet
W20250110 02:51:14.438134 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302af84fd8374ffa96a7503b54849536 already in progress: creating tablet
I20250110 02:51:14.438385 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 0c2412420dc54258a6921ead194c1587 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
W20250110 02:51:14.438630 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d10ace37b96844328ccdb821dad3c5ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d10ace37b96844328ccdb821dad3c5ea already in progress: creating tablet
W20250110 02:51:14.438833 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b43bb790e85343308a1bb1e1110bfb9c already in progress: creating tablet
W20250110 02:51:14.439021 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51adff7f50c4aec94e1892a50ec36c4 already in progress: creating tablet
W20250110 02:51:14.439208 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 46a4745356e3423f9d4892d242da1c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46a4745356e3423f9d4892d242da1c48 already in progress: creating tablet
I20250110 02:51:14.439201 1184 raft_consensus.cc:357] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:14.439365 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f15078fc22f4b32a48c7190b36e2192 already in progress: creating tablet
I20250110 02:51:14.439936 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.440414 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.440461 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.440784 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
W20250110 02:51:14.441147 512 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1edb4e06e40c43438882e1eb622d788e consensus state has no leader
I20250110 02:51:14.441399 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8940c201aa104b0b8bc8d866859fe8ab (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:14.441653 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 344184e1c98d422095cb254f5b2bc792 (table default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]) successfully deleted
I20250110 02:51:14.442484 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.442642 1184 raft_consensus.cc:383] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.442735 1184 raft_consensus.cc:738] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.442977 1184 consensus_queue.cc:260] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.443459 1184 ts_tablet_manager.cc:1428] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20250110 02:51:14.444824 1184 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.446430 1179 ts_tablet_manager.cc:1905] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.457469 1179 ts_tablet_manager.cc:1918] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:14.457619 1184 tablet_bootstrap.cc:654] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.457811 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:14.459051 776 raft_consensus.cc:2463] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.459467 996 leader_election.cc:304] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:14.460050 1467 raft_consensus.cc:2798] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:14.460706 1467 raft_consensus.cc:491] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:14.460824 1467 raft_consensus.cc:3054] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.463444 1467 raft_consensus.cc:513] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.464231 1184 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.464315 1467 leader_election.cc:290] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.464387 1184 ts_tablet_manager.cc:1397] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250110 02:51:14.464654 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:14.465198 994 leader_election.cc:343] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.465198 1184 raft_consensus.cc:357] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.465425 1184 raft_consensus.cc:383] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.465548 1184 raft_consensus.cc:738] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.465545 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00618f611d4c4a84a988125c5d175efc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:14.465806 776 raft_consensus.cc:3054] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.465829 1184 consensus_queue.cc:260] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.466387 1184 ts_tablet_manager.cc:1428] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:14.469511 776 raft_consensus.cc:2463] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:14.470331 996 leader_election.cc:304] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:14.471282 1467 raft_consensus.cc:2798] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:14.471397 1467 raft_consensus.cc:695] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:14.471637 1467 consensus_queue.cc:237] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:14.471748 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 65ae1c19d2dc4778a1cebddbd505f907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ae1c19d2dc4778a1cebddbd505f907 already in progress: creating tablet
W20250110 02:51:14.472004 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2e10e7654cec495bb532fb00e73c020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e10e7654cec495bb532fb00e73c020d already in progress: creating tablet
I20250110 02:51:14.475113 1188 tablet_bootstrap.cc:492] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.475255 1184 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.475373 1188 ts_tablet_manager.cc:1397] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.003s
I20250110 02:51:14.476176 1188 raft_consensus.cc:357] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.476439 1188 raft_consensus.cc:383] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.476562 1188 raft_consensus.cc:738] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.476876 1188 consensus_queue.cc:260] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.478757 1188 ts_tablet_manager.cc:1428] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:14.479131 1180 tablet_replica.cc:331] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.479440 1179 log.cc:1198] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d98b789eaabe4191af40c9f713acb030
I20250110 02:51:14.479532 1180 raft_consensus.cc:2238] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:14.480603 1184 tablet_bootstrap.cc:654] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.482039 1180 pending_rounds.cc:62] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:14.482956 1179 ts_tablet_manager.cc:1939] T d98b789eaabe4191af40c9f713acb030 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.483075 1180 pending_rounds.cc:69] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611734021578752 op_type: NO_OP noop_request { }
I20250110 02:51:14.483217 1180 raft_consensus.cc:2883] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:14.483338 1180 raft_consensus.cc:2267] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.483616 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_08d86300a7254466a9bff50b3f333765" } modify_external_catalogs: true
I20250110 02:51:14.483796 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_08d86300a7254466a9bff50b3f333765" } modify_external_catalogs: true
I20250110 02:51:14.485144 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5" } modify_external_catalogs: true
I20250110 02:51:14.485291 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5" } modify_external_catalogs: true
I20250110 02:51:14.486608 526 master_service.cc:946] ReplaceTablet: received request to replace tablet 019c0d60da5548bb86c93ccd89bf94ff from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.487711 1181 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.487857 1181 ts_tablet_manager.cc:1397] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.003s
I20250110 02:51:14.488113 1180 ts_tablet_manager.cc:1905] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.488585 1181 raft_consensus.cc:357] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.488799 1181 raft_consensus.cc:383] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.488898 1181 raft_consensus.cc:738] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.488991 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:14.489145 1181 consensus_queue.cc:260] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.490098 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 676f5f457d9e4028bfd0b707604c42f5 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.491024 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_380ce069abd543e38d5fe587fcf166cb" } new_table_name: "default.Table_042ec589caf34c7487d9fa394cb51419" modify_external_catalogs: true
I20250110 02:51:14.491259 1181 ts_tablet_manager.cc:1428] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:14.492410 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_65f2cee2d59e485fa3955cff5a8c4bff" } new_table_name: "default.Table_4b25e890ab5a4e4889f28402f205b95f" modify_external_catalogs: true
I20250110 02:51:14.492887 1181 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.493741 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f8df7cc638174115bd8c014aede44d8d" } new_table_name: "default.Table_72d32c146dce4aac925f27638bf13312" modify_external_catalogs: true
I20250110 02:51:14.494902 1181 tablet_bootstrap.cc:654] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.496361 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 9507a5283d8f4d07a293ba474c642298
I20250110 02:51:14.497066 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_161f237741c449f1b10e9b240cca7ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.497517 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_161f237741c449f1b10e9b240cca7ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.498945 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_62d79207c4534af2becc6ccc6da65f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.499483 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d79207c4534af2becc6ccc6da65f1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.502952 1184 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.503098 1184 ts_tablet_manager.cc:1397] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250110 02:51:14.503980 1184 raft_consensus.cc:357] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.504202 1184 raft_consensus.cc:383] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.504278 1184 raft_consensus.cc:738] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.504504 1184 consensus_queue.cc:260] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:14.504885 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 consensus state has no leader
I20250110 02:51:14.505014 1184 ts_tablet_manager.cc:1428] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:14.505100 519 master_service.cc:946] ReplaceTablet: received request to replace tablet d00698a0a9ee4e5da69a745f4a3aa8b4 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.505379 1184 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.504034 1188 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:14.507100 1184 tablet_bootstrap.cc:654] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.510361 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.510808 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.510921 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.511746 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.512465 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.513011 1179 tablet_replica.cc:331] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:14.513084 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9507a5283d8f4d07a293ba474c642298 already in progress: creating tablet
I20250110 02:51:14.513657 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.513659 1179 raft_consensus.cc:2238] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.513865 1179 raft_consensus.cc:2267] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.514556 1179 ts_tablet_manager.cc:1905] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.520638 1179 ts_tablet_manager.cc:1918] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.526093 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d98b789eaabe4191af40c9f713acb030 (table default.Table_2ac26b34e19f4ef2b98ce6410e6b39b5 [id=f3234e13443740dda53c51aecc6cd8da]) successfully deleted
I20250110 02:51:14.526979 1179 log.cc:1198] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2e10e7654cec495bb532fb00e73c020d
I20250110 02:51:14.527500 1179 ts_tablet_manager.cc:1939] T 2e10e7654cec495bb532fb00e73c020d P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:14.527725 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00e97e496b54dcf82b7182591d6f4aa already in progress: creating tablet
W20250110 02:51:14.528280 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 64210.6us
I20250110 02:51:14.529778 1179 tablet_replica.cc:331] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:14.529575 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4042c38de4969889f4821176fc7bb already in progress: creating tablet
I20250110 02:51:14.530205 1179 raft_consensus.cc:2238] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:14.530828 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e25f9643ac0476a90c93fb9aeb921a3 already in progress: creating tablet
I20250110 02:51:14.531250 1184 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.531379 1184 ts_tablet_manager.cc:1397] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20250110 02:51:14.531394 1179 raft_consensus.cc:2267] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.531857 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2e10e7654cec495bb532fb00e73c020d (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]) successfully deleted
I20250110 02:51:14.532163 1184 raft_consensus.cc:357] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.532342 1184 raft_consensus.cc:383] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.532388 1179 ts_tablet_manager.cc:1905] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.532830 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=1 applied=22 ignored=0} inserts{seen=23 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 13 replicates
I20250110 02:51:14.533284 1181 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.533489 1181 ts_tablet_manager.cc:1397] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.005s
I20250110 02:51:14.533567 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Bootstrap complete.
I20250110 02:51:14.534319 1181 raft_consensus.cc:357] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.534680 1181 raft_consensus.cc:383] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.534792 1181 raft_consensus.cc:738] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.535068 1181 consensus_queue.cc:260] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:14.536561 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c09235548b5d4e10a01cbe05d1c94841 consensus state has no leader
I20250110 02:51:14.536880 1181 ts_tablet_manager.cc:1428] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250110 02:51:14.537005 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:14.537168 1181 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:14.537719 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3a9eaf548274b73a5652ba314dab70c consensus state has no leader
I20250110 02:51:14.538424 1512 raft_consensus.cc:491] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.538590 1512 raft_consensus.cc:513] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.539238 1512 leader_election.cc:290] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.539412 517 master_service.cc:946] ReplaceTablet: received request to replace tablet d6fd370138ac4039be55642f1b4c5607 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.539839 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:14.540524 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09fd9734714a4847ab2a4ebc37af8960 consensus state has no leader
W20250110 02:51:14.540653 711 leader_election.cc:343] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.532442 1184 raft_consensus.cc:738] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.540987 1184 consensus_queue.cc:260] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.542806 1181 tablet_bootstrap.cc:654] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.543550 1184 ts_tablet_manager.cc:1428] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:14.543794 1184 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.544899 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a73fa23faec64b8f8178b0dac75d437a" } new_table_name: "default.Table_09dd77d568b8468cbad2375415ecb197" modify_external_catalogs: true
I20250110 02:51:14.546473 1184 tablet_bootstrap.cc:654] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.546535 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d837a8b44265419fa2de3fb6bd251f68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.547039 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d837a8b44265419fa2de3fb6bd251f68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.550506 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.550608 755 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.550905 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.551270 881 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.551789 1180 ts_tablet_manager.cc:1918] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:14.553020 709 leader_election.cc:343] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.553140 709 leader_election.cc:304] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.553495 1512 raft_consensus.cc:2743] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.554863 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f40ec2d4223143dcb1175f357f06de68" } modify_external_catalogs: true
I20250110 02:51:14.555035 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f40ec2d4223143dcb1175f357f06de68" } modify_external_catalogs: true
I20250110 02:51:14.555814 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46" } modify_external_catalogs: true
I20250110 02:51:14.556020 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d00698a0a9ee4e5da69a745f4a3aa8b4
I20250110 02:51:14.555994 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46" } modify_external_catalogs: true
I20250110 02:51:14.556425 519 catalog_manager.cc:6596] ReplaceTablet: tablet d00698a0a9ee4e5da69a745f4a3aa8b4 deleted and replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c
I20250110 02:51:14.557168 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.557583 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.558629 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.558678 1180 log.cc:1198] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/0b74812cb81c44fcb8786cc5dae328e3
I20250110 02:51:14.558843 1178 tablet_replica.cc:331] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.559140 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.559432 1180 ts_tablet_manager.cc:1939] T 0b74812cb81c44fcb8786cc5dae328e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:14.559639 1178 raft_consensus.cc:2238] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.559815 1178 raft_consensus.cc:2267] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.561836 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 0b74812cb81c44fcb8786cc5dae328e3 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]) successfully deleted
I20250110 02:51:14.562254 1179 ts_tablet_manager.cc:1918] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.561486 1556 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Replica starting. Triggering 13 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:14.562454 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:14.562719 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cf711522866d46439d7ee69aa63d7f39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf711522866d46439d7ee69aa63d7f39 already in progress: creating tablet
I20250110 02:51:14.563477 1178 ts_tablet_manager.cc:1905] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:14.564879 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb726d2c9a04963b92a3fb625cf84f3 already in progress: creating tablet
W20250110 02:51:14.565760 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f6df074974974d81a83420e7a3a78070 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df074974974d81a83420e7a3a78070 already in progress: creating tablet
W20250110 02:51:14.566583 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e79175853474cbea951f5a5686bf630 already in progress: creating tablet
W20250110 02:51:14.567409 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b382d634db9642df8e00c5ccaa5787a9 already in progress: creating tablet
I20250110 02:51:14.567883 1180 tablet_replica.cc:331] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:14.568137 1179 log.cc:1198] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:14.568219 1180 raft_consensus.cc:2238] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.568490 1179 ts_tablet_manager.cc:1939] T 3fa76357abae4a95b8fb3af47288e43a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:14.568558 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab302fedec4f4c03989b5d3932dc8d4c has no consensus state
I20250110 02:51:14.568522 1180 raft_consensus.cc:2267] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.569288 1184 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.569439 1184 ts_tablet_manager.cc:1397] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20250110 02:51:14.569434 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 3fa76357abae4a95b8fb3af47288e43a (table default.Table_dce75d4fa88e4bf59fe18cf4725d953f [id=7c8bb5b8a91a4ed9a81403e2e6196406]) successfully deleted
I20250110 02:51:14.570128 1184 raft_consensus.cc:357] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.570338 1184 raft_consensus.cc:383] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.570341 1180 ts_tablet_manager.cc:1905] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.570430 1184 raft_consensus.cc:738] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.570525 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:14.570611 1179 tablet_replica.cc:331] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.570791 1179 raft_consensus.cc:2238] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.570708 1184 consensus_queue.cc:260] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.570945 1179 raft_consensus.cc:2267] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.571211 1184 ts_tablet_manager.cc:1428] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:14.571488 1184 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.571612 1179 ts_tablet_manager.cc:1905] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.572903 517 catalog_manager.cc:6596] ReplaceTablet: tablet d6fd370138ac4039be55642f1b4c5607 deleted and replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726
I20250110 02:51:14.573616 1184 tablet_bootstrap.cc:654] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.575971 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.577056 1181 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.577180 1181 ts_tablet_manager.cc:1397] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
I20250110 02:51:14.577827 1181 raft_consensus.cc:357] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.578045 1181 raft_consensus.cc:383] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.578117 1181 raft_consensus.cc:738] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.578361 1181 consensus_queue.cc:260] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.580108 1106 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:14.580364 1181 ts_tablet_manager.cc:1428] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:14.580857 1181 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.582734 1556 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8d40556dc348e8aa8255d116251a10, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.583508 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:14.588696 1178 ts_tablet_manager.cc:1918] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.588840 1178 log.cc:1198] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f9dbb61a51f14a3f96cc3c77981e8ce5
I20250110 02:51:14.589167 1178 ts_tablet_manager.cc:1939] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.589726 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (table default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]) successfully deleted
I20250110 02:51:14.590056 1178 tablet_replica.cc:331] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:14.590227 1178 raft_consensus.cc:2238] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.590436 1178 raft_consensus.cc:2267] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.591508 1178 ts_tablet_manager.cc:1905] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.592934 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.593725 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.594275 1188 tablet_bootstrap.cc:654] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.595440 1184 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.595574 1184 ts_tablet_manager.cc:1397] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20250110 02:51:14.596786 1184 raft_consensus.cc:357] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.597226 1184 raft_consensus.cc:383] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.597343 1184 raft_consensus.cc:738] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.597568 1184 consensus_queue.cc:260] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.598140 1184 ts_tablet_manager.cc:1428] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:14.599864 1184 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.601091 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:14.606746 1181 tablet_bootstrap.cc:654] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.601927 1184 tablet_bootstrap.cc:654] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.613831 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 28340f0f9a2043a199c74f13755eb702 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:14.616022 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bd6d4c4631994a5792200ba4d4c906f0 (table default.Table_1c00825881c54f55b095b5854ea07eaa [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]). Expected version 2 got 0
I20250110 02:51:14.617301 887 tablet_service.cc:1467] Processing CreateTablet for tablet c5601ca24b8f428e8a51cbeec3664901 (DEFAULT_TABLE table=default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.617311 526 catalog_manager.cc:5526] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:14.617727 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5601ca24b8f428e8a51cbeec3664901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.617883 526 catalog_manager.cc:5526] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:14.618252 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:14.618999 525 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 34acf24a32a54be58d746e86c5be9cfe
I20250110 02:51:14.620860 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bd6d4c4631994a5792200ba4d4c906f0 (table default.Table_1c00825881c54f55b095b5854ea07eaa [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]). Expected version 2 got 0
I20250110 02:51:14.621021 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:14.621133 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_31a8cb16aea94ae7ae6370b386b9dd82 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 1 got 0
I20250110 02:51:14.624861 1179 ts_tablet_manager.cc:1918] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.625038 1179 log.cc:1198] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:14.627921 894 tablet_service.cc:1467] Processing CreateTablet for tablet d761ef0d1e5542b1979c29bb2d579489 (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.628720 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761ef0d1e5542b1979c29bb2d579489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.630096 1023 tablet_service.cc:1467] Processing CreateTablet for tablet c5601ca24b8f428e8a51cbeec3664901 (DEFAULT_TABLE table=default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.631002 748 tablet_service.cc:1467] Processing CreateTablet for tablet d761ef0d1e5542b1979c29bb2d579489 (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.631322 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761ef0d1e5542b1979c29bb2d579489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.631750 1179 ts_tablet_manager.cc:1939] T 019c0d60da5548bb86c93ccd89bf94ff P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.632758 755 tablet_service.cc:1467] Processing CreateTablet for tablet c5601ca24b8f428e8a51cbeec3664901 (DEFAULT_TABLE table=default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.633091 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5601ca24b8f428e8a51cbeec3664901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.634819 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 019c0d60da5548bb86c93ccd89bf94ff (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
I20250110 02:51:14.634913 1037 tablet_service.cc:1467] Processing CreateTablet for tablet d761ef0d1e5542b1979c29bb2d579489 (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.636027 1577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:14.636335 1577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
W20250110 02:51:14.637472 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9507a5283d8f4d07a293ba474c642298 already in progress: creating tablet
I20250110 02:51:14.637653 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 02:51:14.638217 1179 tablet_replica.cc:331] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.638430 1179 raft_consensus.cc:2238] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.638656 1179 raft_consensus.cc:2267] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.640779 1179 ts_tablet_manager.cc:1905] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.640925 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:14.641566 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.641695 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 676f5f457d9e4028bfd0b707604c42f5
I20250110 02:51:14.642035 521 catalog_manager.cc:6596] ReplaceTablet: tablet 676f5f457d9e4028bfd0b707604c42f5 deleted and replaced by tablet 467f87955c4a4d27a6f0483aa109b922
I20250110 02:51:14.642563 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.642899 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 940445840db3441598eba57d3ccb8c9c
I20250110 02:51:14.643754 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.647018 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.647238 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.647747 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.648277 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.649418 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.649446 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.655901 1178 ts_tablet_manager.cc:1918] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.656034 1178 log.cc:1198] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/423c820b81b845719d59a376c18e175a
I20250110 02:51:14.658993 881 tablet_service.cc:1467] Processing CreateTablet for tablet 069d19975e2a4bb0a23b4e9983f411d3 (DEFAULT_TABLE table=default.Table_befac4c75a404c7aa6d15f1fac47ce8b [id=614609c8b3154a4f85cfe592e4f2f182]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.659369 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069d19975e2a4bb0a23b4e9983f411d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.663612 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 28340f0f9a2043a199c74f13755eb702
I20250110 02:51:14.664052 517 catalog_manager.cc:6596] ReplaceTablet: tablet 28340f0f9a2043a199c74f13755eb702 deleted and replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:14.664069 892 tablet_service.cc:1467] Processing CreateTablet for tablet 71534cc7dd014bc1afb365de96977f8c (DEFAULT_TABLE table=default.Table_70a31a6f02a443d68091578355b71adc [id=4946b19488484f39ad47b1118ce03359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.664386 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71534cc7dd014bc1afb365de96977f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.664816 894 tablet_service.cc:1467] Processing CreateTablet for tablet d637597047ab4738a17eca2f5409459a (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.665175 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d637597047ab4738a17eca2f5409459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.666954 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:14.669260 1179 ts_tablet_manager.cc:1918] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.669461 1179 log.cc:1198] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:14.669871 1179 ts_tablet_manager.cc:1939] T cf711522866d46439d7ee69aa63d7f39 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.671828 1178 ts_tablet_manager.cc:1939] T 423c820b81b845719d59a376c18e175a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:14.672792 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 423c820b81b845719d59a376c18e175a (table default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]) successfully deleted
W20250110 02:51:14.674345 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d00698a0a9ee4e5da69a745f4a3aa8b4 already in progress: creating tablet
I20250110 02:51:14.674249 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fbdab52014dddbf63cfef69b09b7b (DEFAULT_TABLE table=default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:14.674911 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6fd370138ac4039be55642f1b4c5607 already in progress: creating tablet
I20250110 02:51:14.675323 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fbdab52014dddbf63cfef69b09b7b (DEFAULT_TABLE table=default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.675546 754 tablet_service.cc:1467] Processing CreateTablet for tablet d637597047ab4738a17eca2f5409459a (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.675706 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7fbdab52014dddbf63cfef69b09b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.675853 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d637597047ab4738a17eca2f5409459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.679072 1030 tablet_service.cc:1467] Processing CreateTablet for tablet d637597047ab4738a17eca2f5409459a (DEFAULT_TABLE table=default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.679668 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dbb61a51f14a3f96cc3c77981e8ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.680202 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c77ec8c7724d5fb408a8bfa53811eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.680662 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 423c820b81b845719d59a376c18e175a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.681128 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.681550 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.682168 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 069d19975e2a4bb0a23b4e9983f411d3 (DEFAULT_TABLE table=default.Table_befac4c75a404c7aa6d15f1fac47ce8b [id=614609c8b3154a4f85cfe592e4f2f182]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:14.682426 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80186dd7d1a240068584b64ce9e9b03f already in progress: creating tablet
I20250110 02:51:14.682996 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.683631 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.684046 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet cf711522866d46439d7ee69aa63d7f39 (table default.Table_08d86300a7254466a9bff50b3f333765 [id=b0585ee264c44482941397f09c2eebb9]) successfully deleted
W20250110 02:51:14.686550 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d00698a0a9ee4e5da69a745f4a3aa8b4 already in progress: creating tablet
I20250110 02:51:14.686635 755 tablet_service.cc:1467] Processing CreateTablet for tablet 069d19975e2a4bb0a23b4e9983f411d3 (DEFAULT_TABLE table=default.Table_befac4c75a404c7aa6d15f1fac47ce8b [id=614609c8b3154a4f85cfe592e4f2f182]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.687021 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069d19975e2a4bb0a23b4e9983f411d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.690155 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676f5f457d9e4028bfd0b707604c42f5 already in progress: creating tablet
W20250110 02:51:14.691166 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940445840db3441598eba57d3ccb8c9c already in progress: creating tablet
W20250110 02:51:14.691370 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28340f0f9a2043a199c74f13755eb702 already in progress: creating tablet
W20250110 02:51:14.691746 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6fd370138ac4039be55642f1b4c5607 already in progress: creating tablet
W20250110 02:51:14.691926 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80186dd7d1a240068584b64ce9e9b03f already in progress: creating tablet
W20250110 02:51:14.692313 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676f5f457d9e4028bfd0b707604c42f5 already in progress: creating tablet
W20250110 02:51:14.692596 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940445840db3441598eba57d3ccb8c9c already in progress: creating tablet
W20250110 02:51:14.692873 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28340f0f9a2043a199c74f13755eb702 already in progress: creating tablet
I20250110 02:51:14.693560 887 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7fbdab52014dddbf63cfef69b09b7b (DEFAULT_TABLE table=default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.694005 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7fbdab52014dddbf63cfef69b09b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.697743 894 tablet_service.cc:1467] Processing CreateTablet for tablet 84adc1fd3f76497b98a0fcbc52a3120a (DEFAULT_TABLE table=default.Table_6a8868f4b22043c193efdd181759b4f4 [id=fd05c3a951c64cbf81dfd168728165f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.697762 892 tablet_service.cc:1467] Processing CreateTablet for tablet c52f74fe375440a28139c27f3f868970 (DEFAULT_TABLE table=default.Table_77792a05daee44cfaaebbe693ef3cd65 [id=f1926d48a14c45699ed056c8b1818b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.698637 881 tablet_service.cc:1467] Processing CreateTablet for tablet 08705d260c0a462b94ba6b0dc5121123 (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.699623 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 71534cc7dd014bc1afb365de96977f8c (DEFAULT_TABLE table=default.Table_70a31a6f02a443d68091578355b71adc [id=4946b19488484f39ad47b1118ce03359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.700820 1039 tablet_service.cc:1467] Processing CreateTablet for tablet c52f74fe375440a28139c27f3f868970 (DEFAULT_TABLE table=default.Table_77792a05daee44cfaaebbe693ef3cd65 [id=f1926d48a14c45699ed056c8b1818b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.702181 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 08705d260c0a462b94ba6b0dc5121123 (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.702817 754 tablet_service.cc:1467] Processing CreateTablet for tablet 08705d260c0a462b94ba6b0dc5121123 (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.703267 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08705d260c0a462b94ba6b0dc5121123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.705612 755 tablet_service.cc:1467] Processing CreateTablet for tablet 71534cc7dd014bc1afb365de96977f8c (DEFAULT_TABLE table=default.Table_70a31a6f02a443d68091578355b71adc [id=4946b19488484f39ad47b1118ce03359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.706583 748 tablet_service.cc:1467] Processing CreateTablet for tablet c52f74fe375440a28139c27f3f868970 (DEFAULT_TABLE table=default.Table_77792a05daee44cfaaebbe693ef3cd65 [id=f1926d48a14c45699ed056c8b1818b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.708112 1481 raft_consensus.cc:1212] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.22->[3.23-3.461] Dedup: 3.35->[3.36-3.461]
I20250110 02:51:14.709048 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84adc1fd3f76497b98a0fcbc52a3120a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.709512 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52f74fe375440a28139c27f3f868970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.711046 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_fe466b4cc96c4d50a4ed42daae488818"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.711527 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe466b4cc96c4d50a4ed42daae488818 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.711836 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1a66990aee93403b8d03464dec9acd0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.712232 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a66990aee93403b8d03464dec9acd0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.713960 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_faf96e0347104a8f828bf7ba504b35ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:14.714594 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_27ec9682796f420ebd7920d531fce2fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.714941 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27ec9682796f420ebd7920d531fce2fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.715227 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_36aaf3ef6c99465eb16fb6565a677486"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.715552 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36aaf3ef6c99465eb16fb6565a677486 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.715804 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_053712a8fc234f22a1cf1caf96485c71" } modify_external_catalogs: true
I20250110 02:51:14.716069 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_01c271ca90d2495b95e92eb25502cff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.716425 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01c271ca90d2495b95e92eb25502cff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.716706 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_26e7f79065c5436eaf670e8a52c72baa" } new_table_name: "default.Table_1e4195192c1b4741bace326bdbff12e6" modify_external_catalogs: true
W20250110 02:51:14.714347 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faf96e0347104a8f828bf7ba504b35ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.719372 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_60665b68321047d3a87e132a42f6c058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.719789 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60665b68321047d3a87e132a42f6c058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.721838 1512 raft_consensus.cc:491] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.723055 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08705d260c0a462b94ba6b0dc5121123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.727072 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 84adc1fd3f76497b98a0fcbc52a3120a (DEFAULT_TABLE table=default.Table_6a8868f4b22043c193efdd181759b4f4 [id=fd05c3a951c64cbf81dfd168728165f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.727643 749 tablet_service.cc:1467] Processing CreateTablet for tablet 84adc1fd3f76497b98a0fcbc52a3120a (DEFAULT_TABLE table=default.Table_6a8868f4b22043c193efdd181759b4f4 [id=fd05c3a951c64cbf81dfd168728165f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:14.733937 1512 raft_consensus.cc:513] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.734805 1512 leader_election.cc:290] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.734977 1512 raft_consensus.cc:491] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.735110 1512 raft_consensus.cc:513] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.735726 1512 leader_election.cc:290] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.736817 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.737455 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5601ca24b8f428e8a51cbeec3664901. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.737649 709 leader_election.cc:343] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.737901 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84adc1fd3f76497b98a0fcbc52a3120a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.738065 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d637597047ab4738a17eca2f5409459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.738426 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761ef0d1e5542b1979c29bb2d579489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.739030 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08705d260c0a462b94ba6b0dc5121123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.738921 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:14.739113 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:14.739445 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52f74fe375440a28139c27f3f868970. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.739533 711 leader_election.cc:343] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.739892 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71534cc7dd014bc1afb365de96977f8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:14.739873 711 leader_election.cc:343] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.740002 711 leader_election.cc:304] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.740446 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7fbdab52014dddbf63cfef69b09b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.740906 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069d19975e2a4bb0a23b4e9983f411d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.741043 1512 raft_consensus.cc:2743] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.742106 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71534cc7dd014bc1afb365de96977f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.755174 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84adc1fd3f76497b98a0fcbc52a3120a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.755208 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3a64d77e95f7457490d8a6641411e1dc" } new_table_name: "default.Table_c4155bb1db484441b38ef15357e8f40f" modify_external_catalogs: true
I20250110 02:51:14.761961 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c52f74fe375440a28139c27f3f868970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:14.737257 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.767916 1188 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.768589 1188 ts_tablet_manager.cc:1397] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.265s user 0.005s sys 0.001s
I20250110 02:51:14.769291 1188 raft_consensus.cc:357] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.769492 1188 raft_consensus.cc:383] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.769551 1188 raft_consensus.cc:738] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.769776 1188 consensus_queue.cc:260] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.770263 1188 ts_tablet_manager.cc:1428] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:14.771868 709 leader_election.cc:343] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.772014 709 leader_election.cc:304] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.773164 1512 raft_consensus.cc:2743] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.773658 1188 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:14.781548 1188 tablet_bootstrap.cc:654] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.806949 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_053712a8fc234f22a1cf1caf96485c71" } modify_external_catalogs: true
I20250110 02:51:14.815244 1588 raft_consensus.cc:491] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.815470 1588 raft_consensus.cc:513] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.816351 1588 leader_election.cc:290] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.816946 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:14.817497 709 leader_election.cc:343] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.823714 1188 tablet_bootstrap.cc:492] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.823920 1188 ts_tablet_manager.cc:1397] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20250110 02:51:14.824774 1188 raft_consensus.cc:357] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.825034 1188 raft_consensus.cc:383] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.825198 1188 raft_consensus.cc:738] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.825546 1188 consensus_queue.cc:260] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.825796 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:14.826133 1188 ts_tablet_manager.cc:1428] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:14.826565 1188 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:14.826593 1467 raft_consensus.cc:491] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:14.827379 711 leader_election.cc:343] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.827548 711 leader_election.cc:304] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.828330 1188 tablet_bootstrap.cc:654] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.828480 1588 raft_consensus.cc:2743] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.828823 1467 raft_consensus.cc:513] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.829910 1467 leader_election.cc:290] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:14.832070 996 leader_election.cc:343] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:14.834151 1184 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.834407 1184 ts_tablet_manager.cc:1397] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.235s user 0.014s sys 0.068s
I20250110 02:51:14.835243 1184 raft_consensus.cc:357] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.835567 1184 raft_consensus.cc:383] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.835686 1184 raft_consensus.cc:738] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.835999 1184 consensus_queue.cc:260] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.836589 1184 ts_tablet_manager.cc:1428] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:14.842742 1184 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.845433 1184 tablet_bootstrap.cc:654] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.830708 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af6ec40c7a4442b91680d8125c2d48" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.848362 917 raft_consensus.cc:2463] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:14.848942 994 leader_election.cc:304] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:14.849831 1467 raft_consensus.cc:2798] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:14.849939 1467 raft_consensus.cc:491] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:14.850075 1588 raft_consensus.cc:491] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.850143 1467 raft_consensus.cc:3054] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.850219 1588 raft_consensus.cc:513] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.851016 1588 leader_election.cc:290] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:14.851267 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:14.851756 709 leader_election.cc:343] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.853730 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:14.855788 711 leader_election.cc:343] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.855957 711 leader_election.cc:304] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:14.857270 1184 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.857414 1184 ts_tablet_manager.cc:1397] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20250110 02:51:14.858176 1184 raft_consensus.cc:357] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.860118 1184 raft_consensus.cc:383] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.860240 1184 raft_consensus.cc:738] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.860507 1184 consensus_queue.cc:260] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.860872 1512 raft_consensus.cc:2743] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.861205 1184 ts_tablet_manager.cc:1428] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250110 02:51:14.861586 1184 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.871891 1184 tablet_bootstrap.cc:654] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.876237 1467 raft_consensus.cc:513] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.877776 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af6ec40c7a4442b91680d8125c2d48" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:14.878028 917 raft_consensus.cc:3054] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:14.879877 1184 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.880034 1184 ts_tablet_manager.cc:1397] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20250110 02:51:14.881006 917 raft_consensus.cc:2463] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:14.881318 1184 raft_consensus.cc:357] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.881633 994 leader_election.cc:304] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:14.881567 1184 raft_consensus.cc:383] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.881829 1184 raft_consensus.cc:738] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.881923 1386 raft_consensus.cc:2798] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:14.882318 1184 consensus_queue.cc:260] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.882859 1184 ts_tablet_manager.cc:1428] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:14.883181 1386 raft_consensus.cc:695] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:14.883474 1386 consensus_queue.cc:237] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.886358 1184 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.888633 1184 tablet_bootstrap.cc:654] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:14.890100 996 leader_election.cc:343] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:14.890406 1467 leader_election.cc:290] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:14.907573 1184 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.907936 1184 ts_tablet_manager.cc:1397] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20250110 02:51:14.909003 1184 raft_consensus.cc:357] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.909374 1184 raft_consensus.cc:383] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.909497 1184 raft_consensus.cc:738] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.909778 1184 consensus_queue.cc:260] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.910630 1184 ts_tablet_manager.cc:1428] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250110 02:51:14.911288 1181 tablet_bootstrap.cc:492] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.911464 1181 ts_tablet_manager.cc:1397] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.331s user 0.019s sys 0.074s
I20250110 02:51:14.912349 1181 raft_consensus.cc:357] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.912600 1181 raft_consensus.cc:383] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.912664 1181 raft_consensus.cc:738] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.912940 1181 consensus_queue.cc:260] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.913568 1181 ts_tablet_manager.cc:1428] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:14.913869 1181 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.915526 1181 tablet_bootstrap.cc:654] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.915524 1184 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.917479 1184 tablet_bootstrap.cc:654] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.920711 1181 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.920920 1181 ts_tablet_manager.cc:1397] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250110 02:51:14.921885 1181 raft_consensus.cc:357] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.923887 1181 raft_consensus.cc:383] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.924014 1181 raft_consensus.cc:738] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.924234 1181 consensus_queue.cc:260] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.924793 1181 ts_tablet_manager.cc:1428] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250110 02:51:14.925145 1181 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.927294 1181 tablet_bootstrap.cc:654] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:14.933784 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e03d53860ab744968693cf96467c3a19 consensus state has no leader
I20250110 02:51:14.938970 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:14.939337 1184 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.939519 1184 ts_tablet_manager.cc:1397] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20250110 02:51:14.939743 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5bf91a5e0e004e1880fdf025d5d9ba38" } modify_external_catalogs: true
I20250110 02:51:14.939927 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5bf91a5e0e004e1880fdf025d5d9ba38" } modify_external_catalogs: true
I20250110 02:51:14.939985 517 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:14.940611 1184 raft_consensus.cc:357] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.940718 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a792a53d593141ddb875604e99d221d1" } new_table_name: "default.Table_2e282311da134ac6abccffab8398357b" modify_external_catalogs: true
I20250110 02:51:14.940824 1184 raft_consensus.cc:383] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.940884 1184 raft_consensus.cc:738] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.940840 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_71d270ae71ca41e384eb525387ffd768" } new_table_name: "default.Table_6f66870093e04c74a3762a4c14c6bc91" modify_external_catalogs: true
I20250110 02:51:14.941691 1184 consensus_queue.cc:260] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.942298 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:14.942672 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.942615 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_06ee6a222f9f4105a670d8943bf209bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.943081 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ee6a222f9f4105a670d8943bf209bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.943116 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:14.943328 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9289de0630804f9aa60fdaf5f3f15f68 already in progress: creating tablet
W20250110 02:51:14.943986 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9289de0630804f9aa60fdaf5f3f15f68 already in progress: creating tablet
I20250110 02:51:14.945091 1184 ts_tablet_manager.cc:1428] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250110 02:51:14.945354 1184 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.945585 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_48bb129a959843fcbad0b8d6344c6660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.946018 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48bb129a959843fcbad0b8d6344c6660 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.944700 1386 raft_consensus.cc:491] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:14.946693 1386 raft_consensus.cc:513] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.947013 1184 tablet_bootstrap.cc:654] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:14.947944 996 leader_election.cc:343] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:14.948272 748 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.948520 1179 tablet_replica.cc:331] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.948711 1179 raft_consensus.cc:2238] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.948906 1179 raft_consensus.cc:2267] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.949265 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9f920cde954f31acebc6d0b6965e58" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:14.949609 1179 ts_tablet_manager.cc:1905] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:14.949781 526 master_service.cc:946] ReplaceTablet: received request to replace tablet d20177229f7b4f11875e0007afb1dd00 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:14.949772 994 leader_election.cc:343] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:14.949896 994 leader_election.cc:304] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:14.950140 1467 raft_consensus.cc:2743] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:14.950251 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_90e43feb19a343f6b3c9a20876ec8c23" } modify_external_catalogs: true
I20250110 02:51:14.950381 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_90e43feb19a343f6b3c9a20876ec8c23" } modify_external_catalogs: true
I20250110 02:51:14.950853 1386 leader_election.cc:290] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:14.952580 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0e2df09ed397432f8697c6baf59aaa53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:14.953018 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e2df09ed397432f8697c6baf59aaa53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:14.954092 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1c00825881c54f55b095b5854ea07eaa" } new_table_name: "default.Table_742e66ce36c0463a978cb00de2f01135" modify_external_catalogs: true
I20250110 02:51:14.954702 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
W20250110 02:51:14.957253 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae3033595f824affb2d300c3964e1e8e has no consensus state
I20250110 02:51:14.957551 1181 tablet_bootstrap.cc:492] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:14.957729 1181 ts_tablet_manager.cc:1397] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.003s
I20250110 02:51:14.958472 1181 raft_consensus.cc:357] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.958707 1181 raft_consensus.cc:383] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.958782 1181 raft_consensus.cc:738] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.958985 1181 consensus_queue.cc:260] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.959455 1181 ts_tablet_manager.cc:1428] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:14.959789 1181 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:14.961386 1181 tablet_bootstrap.cc:654] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.963165 1184 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:14.963317 1184 ts_tablet_manager.cc:1397] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250110 02:51:14.968441 1184 raft_consensus.cc:357] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.968694 1184 raft_consensus.cc:383] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.968793 1184 raft_consensus.cc:738] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.969671 1184 consensus_queue.cc:260] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:14.965108 1179 ts_tablet_manager.cc:1918] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:14.970652 1179 log.cc:1198] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:14.971174 1179 ts_tablet_manager.cc:1939] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:14.972524 1180 ts_tablet_manager.cc:1918] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:14.972648 1180 log.cc:1198] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d98b789eaabe4191af40c9f713acb030
I20250110 02:51:14.973021 1180 ts_tablet_manager.cc:1939] T d98b789eaabe4191af40c9f713acb030 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:14.973352 1184 ts_tablet_manager.cc:1428] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20250110 02:51:14.974705 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (table default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]) successfully deleted
W20250110 02:51:14.975023 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00ebf8debb9d4394839d1c1cb97820a5 consensus state has no leader
W20250110 02:51:14.977386 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 452223af5f6a4836a0f582d394a9453e consensus state has no leader
I20250110 02:51:14.981967 1184 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:14.983886 1184 tablet_bootstrap.cc:654] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.984709 1188 tablet_bootstrap.cc:492] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:14.984956 1188 ts_tablet_manager.cc:1397] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.158s user 0.006s sys 0.014s
I20250110 02:51:14.985723 1188 raft_consensus.cc:357] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.985997 1188 raft_consensus.cc:383] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:14.986095 1188 raft_consensus.cc:738] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:14.986423 1188 consensus_queue.cc:260] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:14.987143 1188 ts_tablet_manager.cc:1428] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:14.987957 1188 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:14.988379 963 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:14.990458 1188 tablet_bootstrap.cc:654] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:14.992551 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d837a8b44265419fa2de3fb6bd251f68" } modify_external_catalogs: true
I20250110 02:51:14.992738 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d837a8b44265419fa2de3fb6bd251f68" } modify_external_catalogs: true
I20250110 02:51:14.993062 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.993113 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.993309 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.993402 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:14.993472 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_bf5730ffdb414255925a1360ec7d7626" } modify_external_catalogs: true
I20250110 02:51:14.993614 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_bf5730ffdb414255925a1360ec7d7626" } modify_external_catalogs: true
I20250110 02:51:14.993809 1179 tablet_replica.cc:331] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:14.994050 1179 raft_consensus.cc:2238] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:14.994268 1179 raft_consensus.cc:2267] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:14.999253 1179 ts_tablet_manager.cc:1905] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:15.006975 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4042c38de4969889f4821176fc7bb already in progress: creating tablet
W20250110 02:51:15.007223 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00e97e496b54dcf82b7182591d6f4aa already in progress: creating tablet
W20250110 02:51:15.007416 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9507a5283d8f4d07a293ba474c642298 already in progress: creating tablet
W20250110 02:51:15.007637 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cf711522866d46439d7ee69aa63d7f39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf711522866d46439d7ee69aa63d7f39 already in progress: creating tablet
I20250110 02:51:15.009102 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d970e2fe72b4727968ca605b3e772cc from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:15.009559 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d00698a0a9ee4e5da69a745f4a3aa8b4 already in progress: creating tablet
I20250110 02:51:15.009828 523 master_service.cc:946] ReplaceTablet: received request to replace tablet 08705d260c0a462b94ba6b0dc5121123 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:15.010185 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6fd370138ac4039be55642f1b4c5607 already in progress: creating tablet
W20250110 02:51:15.010890 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80186dd7d1a240068584b64ce9e9b03f already in progress: creating tablet
I20250110 02:51:15.011607 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 940445840db3441598eba57d3ccb8c9c from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:15.011662 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676f5f457d9e4028bfd0b707604c42f5 already in progress: creating tablet
I20250110 02:51:15.012068 520 master_service.cc:946] ReplaceTablet: received request to replace tablet f6df074974974d81a83420e7a3a78070 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:15.012471 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940445840db3441598eba57d3ccb8c9c already in progress: creating tablet
I20250110 02:51:15.012943 1180 tablet_replica.cc:331] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.013229 1180 raft_consensus.cc:2238] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.013334 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.013463 1180 raft_consensus.cc:2267] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.015621 1179 ts_tablet_manager.cc:1918] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.016304 1179 log.cc:1198] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:15.016673 1179 ts_tablet_manager.cc:1939] T 7d4422eca5604d149e73c2a669d07af0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.017817 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7d4422eca5604d149e73c2a669d07af0 (table default.Table_0d23d32f6ce84a0594f62a4ca18dcc62 [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
I20250110 02:51:15.018190 1588 raft_consensus.cc:491] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.018345 1588 raft_consensus.cc:513] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.018445 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_69e74c02a7094d698547edc4e145bdd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.019022 1588 leader_election.cc:290] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:15.019093 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69e74c02a7094d698547edc4e145bdd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.019845 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e744cdc1fcb94d46b48b684763ecf1bf" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:15.021027 709 leader_election.cc:343] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:15.021502 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd6d4c4631994a5792200ba4d4c906f0 consensus state has no leader
I20250110 02:51:15.014161 1180 ts_tablet_manager.cc:1905] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.019050 1181 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.020520 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e744cdc1fcb94d46b48b684763ecf1bf" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.022082 1181 ts_tablet_manager.cc:1397] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.013s
W20250110 02:51:15.022526 711 leader_election.cc:343] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.022687 711 leader_election.cc:304] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.024353 522 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:15.025401 1184 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.025558 1184 ts_tablet_manager.cc:1397] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250110 02:51:15.025652 521 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:15.026510 1588 raft_consensus.cc:2743] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.026410 1184 raft_consensus.cc:357] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.027006 1184 raft_consensus.cc:383] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.027093 1184 raft_consensus.cc:738] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.027307 1184 consensus_queue.cc:260] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.027825 1184 ts_tablet_manager.cc:1428] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:15.030068 887 tablet_service.cc:1467] Processing CreateTablet for tablet b010e9c549214937889fa281542e752e (DEFAULT_TABLE table=default.Table_62d79207c4534af2becc6ccc6da65f1e [id=8e8cecbb687a48009172ae19382c9b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.030524 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b010e9c549214937889fa281542e752e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.033349 1184 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.034639 881 tablet_service.cc:1467] Processing CreateTablet for tablet dba0cb7970f1453dac256b255ed7cac0 (DEFAULT_TABLE table=default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.034968 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba0cb7970f1453dac256b255ed7cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.036219 1184 tablet_bootstrap.cc:654] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.036638 1181 raft_consensus.cc:357] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.036926 1181 raft_consensus.cc:383] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.037032 1181 raft_consensus.cc:738] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.037230 1179 tablet_replica.cc:331] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.037751 1179 raft_consensus.cc:2238] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.037952 1179 raft_consensus.cc:2267] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.038677 892 tablet_service.cc:1467] Processing CreateTablet for tablet d3858638b6fc48d0bb8b8f97370a791d (DEFAULT_TABLE table=default.Table_8f8cd5df24f147c5ae832efed9aad406 [id=0861a4be8cc1428285ab917dc1a71f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.039021 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3858638b6fc48d0bb8b8f97370a791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.039651 1179 ts_tablet_manager.cc:1905] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.042502 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af6ec40c7a4442b91680d8125c2d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.042886 888 tablet_service.cc:1467] Processing CreateTablet for tablet b25bbf8aa71e478587093627c715310f (DEFAULT_TABLE table=default.Table_a3ba2bc51a3b4fc6b6451349e08b106d [id=14dfd1504b024d6bbd154720e1e8c43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.043284 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25bbf8aa71e478587093627c715310f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.046298 1179 ts_tablet_manager.cc:1918] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.046274 1181 consensus_queue.cc:260] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.046617 882 tablet_service.cc:1467] Processing CreateTablet for tablet 718d3add9d0a4e4b8633f2c977e316e6 (DEFAULT_TABLE table=default.Table_c99830786e0e4b7b956e604dd3a5fcc0 [id=2af0614c37b44ebb9baa17b28b82f381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.046985 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718d3add9d0a4e4b8633f2c977e316e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.047046 1181 ts_tablet_manager.cc:1428] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250110 02:51:15.049960 896 tablet_service.cc:1467] Processing CreateTablet for tablet 39cfc508c1b84299a2f16d307bb25354 (DEFAULT_TABLE table=default.Table_78b575ae2603466d9703e8780d3cd413 [id=a14528ef7bd24ede9db29c29df81c11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.050240 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cfc508c1b84299a2f16d307bb25354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.052354 1179 log.cc:1198] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/585320c0970c4097b195caaae33789b5
I20250110 02:51:15.052753 1179 ts_tablet_manager.cc:1939] T 585320c0970c4097b195caaae33789b5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.053346 1181 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.054698 1178 tablet_replica.cc:331] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:15.054890 1178 raft_consensus.cc:2238] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.055063 1178 raft_consensus.cc:2267] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.055640 1181 tablet_bootstrap.cc:654] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.056161 1179 tablet_replica.cc:331] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.056376 1179 raft_consensus.cc:2238] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.057205 1179 raft_consensus.cc:2267] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.058251 1179 ts_tablet_manager.cc:1905] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.059209 1178 ts_tablet_manager.cc:1905] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.067703 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 585320c0970c4097b195caaae33789b5 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
I20250110 02:51:15.068504 1386 raft_consensus.cc:491] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.068684 1035 tablet_service.cc:1467] Processing CreateTablet for tablet d3858638b6fc48d0bb8b8f97370a791d (DEFAULT_TABLE table=default.Table_8f8cd5df24f147c5ae832efed9aad406 [id=0861a4be8cc1428285ab917dc1a71f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.068683 1386 raft_consensus.cc:513] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.069103 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3858638b6fc48d0bb8b8f97370a791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.069159 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b010e9c549214937889fa281542e752e (DEFAULT_TABLE table=default.Table_62d79207c4534af2becc6ccc6da65f1e [id=8e8cecbb687a48009172ae19382c9b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.069418 1386 leader_election.cc:290] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.069730 1031 tablet_service.cc:1467] Processing CreateTablet for tablet dba0cb7970f1453dac256b255ed7cac0 (DEFAULT_TABLE table=default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.070268 1023 tablet_service.cc:1467] Processing CreateTablet for tablet b25bbf8aa71e478587093627c715310f (DEFAULT_TABLE table=default.Table_a3ba2bc51a3b4fc6b6451349e08b106d [id=14dfd1504b024d6bbd154720e1e8c43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.070619 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 718d3add9d0a4e4b8633f2c977e316e6 (DEFAULT_TABLE table=default.Table_c99830786e0e4b7b956e604dd3a5fcc0 [id=2af0614c37b44ebb9baa17b28b82f381]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:15.070969 996 leader_election.cc:343] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 4e2c01af512f460daa98b19444f72172
I20250110 02:51:15.069450 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b010e9c549214937889fa281542e752e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.071185 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2c01af512f460daa98b19444f72172 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.071389 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 514704cd935b44e8b760221224332509 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.072600 748 tablet_service.cc:1467] Processing CreateTablet for tablet b010e9c549214937889fa281542e752e (DEFAULT_TABLE table=default.Table_62d79207c4534af2becc6ccc6da65f1e [id=8e8cecbb687a48009172ae19382c9b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.071038 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.073608 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25bbf8aa71e478587093627c715310f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.074460 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b010e9c549214937889fa281542e752e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:15.077764 994 leader_election.cc:343] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 4e2c01af512f460daa98b19444f72172
I20250110 02:51:15.077785 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718d3add9d0a4e4b8633f2c977e316e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.080596 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba0cb7970f1453dac256b255ed7cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.082422 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:15.083091 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 08705d260c0a462b94ba6b0dc5121123
I20250110 02:51:15.083618 523 catalog_manager.cc:6596] ReplaceTablet: tablet 08705d260c0a462b94ba6b0dc5121123 deleted and replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a
I20250110 02:51:15.083786 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet f9b379b357194d729108428cbaebd51a
I20250110 02:51:15.085791 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:15.086311 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 95b14d7592024a34baf7756d92e546d9
I20250110 02:51:15.088642 1181 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.088958 1039 tablet_service.cc:1467] Processing CreateTablet for tablet f9b379b357194d729108428cbaebd51a (DEFAULT_TABLE table=default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.089133 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 39cfc508c1b84299a2f16d307bb25354 (DEFAULT_TABLE table=default.Table_78b575ae2603466d9703e8780d3cd413 [id=a14528ef7bd24ede9db29c29df81c11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.088604 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.089392 1180 ts_tablet_manager.cc:1918] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.089556 1180 log.cc:1198] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f9dbb61a51f14a3f96cc3c77981e8ce5
I20250110 02:51:15.079988 994 leader_election.cc:304] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:15.093221 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.093606 1181 ts_tablet_manager.cc:1397] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.004s
I20250110 02:51:15.093818 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.094080 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ab302fedec4f4c03989b5d3932dc8d4c (DEFAULT_TABLE table=default.Table_09dd77d568b8468cbad2375415ecb197 [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.094511 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.095186 894 tablet_service.cc:1467] Processing CreateTablet for tablet f9b379b357194d729108428cbaebd51a (DEFAULT_TABLE table=default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.095574 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b379b357194d729108428cbaebd51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.099992 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.100383 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.100577 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.101146 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.101370 879 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.100948 1181 raft_consensus.cc:357] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.102119 897 tablet_service.cc:1467] Processing CreateTablet for tablet bb595013e809480a814cb9c7aed856e6 (DEFAULT_TABLE table=default.Table_1d2e132b16324d3ea79f29a7339f4d5e [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.102427 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb595013e809480a814cb9c7aed856e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.102705 1181 raft_consensus.cc:383] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.102810 1181 raft_consensus.cc:738] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.103032 885 tablet_service.cc:1467] Processing CreateTablet for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (DEFAULT_TABLE table=default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a [id=38abb97bc5e34b318c9a9081c1cdc7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.101293 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (DEFAULT_TABLE table=default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a [id=38abb97bc5e34b318c9a9081c1cdc7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.103506 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b379b357194d729108428cbaebd51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.103658 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.104350 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.104323 1181 consensus_queue.cc:260] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.104758 891 tablet_service.cc:1467] Processing CreateTablet for tablet 71ffa250a3554a7b8f3a9c34ee7e4674 (DEFAULT_TABLE table=default.Table_6f9765add6c947e8b8154e36e6d358dc [id=7badf15e8d1543209edcffa451e3b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.104998 1181 ts_tablet_manager.cc:1428] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250110 02:51:15.105068 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ffa250a3554a7b8f3a9c34ee7e4674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.106678 1181 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.108304 1037 tablet_service.cc:1467] Processing CreateTablet for tablet bb595013e809480a814cb9c7aed856e6 (DEFAULT_TABLE table=default.Table_1d2e132b16324d3ea79f29a7339f4d5e [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.109011 889 tablet_service.cc:1467] Processing CreateTablet for tablet ab302fedec4f4c03989b5d3932dc8d4c (DEFAULT_TABLE table=default.Table_09dd77d568b8468cbad2375415ecb197 [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.109337 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab302fedec4f4c03989b5d3932dc8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.110836 1467 raft_consensus.cc:491] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.110970 1467 raft_consensus.cc:513] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:15.117473 994 leader_election.cc:343] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 423c820b81b845719d59a376c18e175a
I20250110 02:51:15.120116 890 tablet_service.cc:1467] Processing CreateTablet for tablet 467f87955c4a4d27a6f0483aa109b922 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.121608 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467f87955c4a4d27a6f0483aa109b922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.128083 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cfc508c1b84299a2f16d307bb25354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.128053 1386 raft_consensus.cc:2743] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.128513 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab302fedec4f4c03989b5d3932dc8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.128616 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.129391 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.130168 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb595013e809480a814cb9c7aed856e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.132062 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 71ffa250a3554a7b8f3a9c34ee7e4674 (DEFAULT_TABLE table=default.Table_6f9765add6c947e8b8154e36e6d358dc [id=7badf15e8d1543209edcffa451e3b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.132653 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ffa250a3554a7b8f3a9c34ee7e4674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.132308 1467 leader_election.cc:290] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.133401 1180 ts_tablet_manager.cc:1939] T f9dbb61a51f14a3f96cc3c77981e8ce5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.134435 1180 tablet_replica.cc:331] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.134783 1180 raft_consensus.cc:2238] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.160256 1178 ts_tablet_manager.cc:1918] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.160423 1178 log.cc:1198] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:15.160781 1178 ts_tablet_manager.cc:1939] T 70af6ec40c7a4442b91680d8125c2d48 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:15.162655 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f9dbb61a51f14a3f96cc3c77981e8ce5 (table default.Table_ff78b6f4c98a42b69c23212d1fa6f09b [id=2612c0ab6b944ff184a651674b1cb6e9]) successfully deleted
I20250110 02:51:15.163117 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 70af6ec40c7a4442b91680d8125c2d48 (table default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]) successfully deleted
W20250110 02:51:15.164809 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d970e2fe72b4727968ca605b3e772cc already in progress: creating tablet
W20250110 02:51:15.165086 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08705d260c0a462b94ba6b0dc5121123 already in progress: creating tablet
I20250110 02:51:15.165367 755 tablet_service.cc:1467] Processing CreateTablet for tablet d3858638b6fc48d0bb8b8f97370a791d (DEFAULT_TABLE table=default.Table_8f8cd5df24f147c5ae832efed9aad406 [id=0861a4be8cc1428285ab917dc1a71f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.165982 1588 raft_consensus.cc:491] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.166011 749 tablet_service.cc:1467] Processing CreateTablet for tablet dba0cb7970f1453dac256b255ed7cac0 (DEFAULT_TABLE table=default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.165369 754 tablet_service.cc:1467] Processing CreateTablet for tablet b25bbf8aa71e478587093627c715310f (DEFAULT_TABLE table=default.Table_a3ba2bc51a3b4fc6b6451349e08b106d [id=14dfd1504b024d6bbd154720e1e8c43f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.166800 1588 raft_consensus.cc:513] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.166935 745 tablet_service.cc:1467] Processing CreateTablet for tablet 718d3add9d0a4e4b8633f2c977e316e6 (DEFAULT_TABLE table=default.Table_c99830786e0e4b7b956e604dd3a5fcc0 [id=2af0614c37b44ebb9baa17b28b82f381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.167726 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3858638b6fc48d0bb8b8f97370a791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.167757 753 tablet_service.cc:1467] Processing CreateTablet for tablet 39cfc508c1b84299a2f16d307bb25354 (DEFAULT_TABLE table=default.Table_78b575ae2603466d9703e8780d3cd413 [id=a14528ef7bd24ede9db29c29df81c11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.168112 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cfc508c1b84299a2f16d307bb25354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.169512 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718d3add9d0a4e4b8633f2c977e316e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.169811 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a4745356e3423f9d4892d242da1c48" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.170388 1588 leader_election.cc:290] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.170408 746 tablet_service.cc:1467] Processing CreateTablet for tablet f9b379b357194d729108428cbaebd51a (DEFAULT_TABLE table=default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.170751 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25bbf8aa71e478587093627c715310f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.170995 1188 tablet_bootstrap.cc:492] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:15.171125 1188 ts_tablet_manager.cc:1397] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.183s user 0.011s sys 0.009s
I20250110 02:51:15.171855 1188 raft_consensus.cc:357] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.172178 1188 raft_consensus.cc:383] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.172297 1188 raft_consensus.cc:738] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.172538 1188 consensus_queue.cc:260] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.173197 1188 ts_tablet_manager.cc:1428] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:15.173523 1188 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:15.174443 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.175678 709 leader_election.cc:343] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.176246 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b379b357194d729108428cbaebd51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.149066 1467 raft_consensus.cc:491] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.178164 1467 raft_consensus.cc:513] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.178789 1467 leader_election.cc:290] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.179625 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:15.181957 994 leader_election.cc:343] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.160768 1181 tablet_bootstrap.cc:654] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:15.175029 996 leader_election.cc:343] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 423c820b81b845719d59a376c18e175a
I20250110 02:51:15.182569 996 leader_election.cc:304] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:15.175384 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a4745356e3423f9d4892d242da1c48" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.183568 1467 raft_consensus.cc:2743] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.183722 1180 raft_consensus.cc:2267] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.183748 1467 consensus_queue.cc:1035] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:15.184136 1188 tablet_bootstrap.cc:654] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.184264 1180 ts_tablet_manager.cc:1905] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.187407 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba0cb7970f1453dac256b255ed7cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.191404 1188 tablet_bootstrap.cc:492] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:15.191634 1188 ts_tablet_manager.cc:1397] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250110 02:51:15.192723 1188 raft_consensus.cc:357] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.193046 1188 raft_consensus.cc:383] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.193169 1188 raft_consensus.cc:738] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.193526 1188 consensus_queue.cc:260] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.194401 1188 ts_tablet_manager.cc:1428] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:15.194705 1188 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:15.196189 1188 tablet_bootstrap.cc:654] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.196295 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:15.196492 776 raft_consensus.cc:2463] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:15.197027 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.197091 711 leader_election.cc:343] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.197443 744 tablet_service.cc:1467] Processing CreateTablet for tablet ab302fedec4f4c03989b5d3932dc8d4c (DEFAULT_TABLE table=default.Table_09dd77d568b8468cbad2375415ecb197 [id=c6fdcf73aa2a471faeaca5a4a8f70258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.197907 711 leader_election.cc:304] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.198297 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.198925 1588 raft_consensus.cc:2743] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.205559 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.214475 1181 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.214622 1181 ts_tablet_manager.cc:1397] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.003s
I20250110 02:51:15.215129 1188 tablet_bootstrap.cc:492] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:15.215312 1188 ts_tablet_manager.cc:1397] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.006s
I20250110 02:51:15.215381 1181 raft_consensus.cc:357] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.215567 1181 raft_consensus.cc:383] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.215656 1181 raft_consensus.cc:738] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.218839 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_50adace768fa44c698737e7843f5ac75" } new_table_name: "default.Table_46ddbd83b65347bba213571663209760" modify_external_catalogs: true
I20250110 02:51:15.219832 996 leader_election.cc:304] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:15.221249 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a73b4a8bd4a741d68bae45ba1bd417e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.221499 1386 raft_consensus.cc:2798] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:15.221601 1386 raft_consensus.cc:491] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:15.221685 1386 raft_consensus.cc:3054] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:15.221725 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:15.222615 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
W20250110 02:51:15.222882 510 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 57579.9us
W20250110 02:51:15.223194 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet da8f247c862e4cb68452e230857ee6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da8f247c862e4cb68452e230857ee6b0 already in progress: creating tablet
W20250110 02:51:15.223469 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 95b14d7592024a34baf7756d92e546d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95b14d7592024a34baf7756d92e546d9 already in progress: creating tablet
I20250110 02:51:15.226365 1386 raft_consensus.cc:513] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.227900 1386 leader_election.cc:290] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.228538 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:15.229112 994 leader_election.cc:343] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.230000 753 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.230149 752 tablet_service.cc:1467] Processing CreateTablet for tablet bb595013e809480a814cb9c7aed856e6 (DEFAULT_TABLE table=default.Table_1d2e132b16324d3ea79f29a7339f4d5e [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.230875 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab302fedec4f4c03989b5d3932dc8d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.232933 1179 ts_tablet_manager.cc:1918] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.233094 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc0532c94eb401e9d299e79556b4cf1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:15.233253 775 raft_consensus.cc:3054] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.234301 745 tablet_service.cc:1467] Processing CreateTablet for tablet 71ffa250a3554a7b8f3a9c34ee7e4674 (DEFAULT_TABLE table=default.Table_6f9765add6c947e8b8154e36e6d358dc [id=7badf15e8d1543209edcffa451e3b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.234534 740 tablet_service.cc:1467] Processing CreateTablet for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (DEFAULT_TABLE table=default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a [id=38abb97bc5e34b318c9a9081c1cdc7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.235482 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0d23d32f6ce84a0594f62a4ca18dcc62" } new_table_name: "default.Table_5594db9942c346058f56f1abc8b3262d" modify_external_catalogs: true
I20250110 02:51:15.236173 1184 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.236070 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_adb6100fd2ea489590831f0508f53e3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.236547 1179 log.cc:1198] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f14f06ba7d0f44418693e36f6bc6d888
W20250110 02:51:15.236582 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adb6100fd2ea489590831f0508f53e3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.236963 1179 ts_tablet_manager.cc:1939] T f14f06ba7d0f44418693e36f6bc6d888 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.237823 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_ce235042d71840c0a05ae04e253c9517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.238363 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce235042d71840c0a05ae04e253c9517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.239472 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e04ee12d5f244793a7a44387a6243c1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.239817 753 tablet_service.cc:1467] Processing CreateTablet for tablet 467f87955c4a4d27a6f0483aa109b922 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:15.239886 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e04ee12d5f244793a7a44387a6243c1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.240082 1184 ts_tablet_manager.cc:1397] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.203s user 0.029s sys 0.076s
I20250110 02:51:15.240759 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ffa250a3554a7b8f3a9c34ee7e4674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.240875 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_93c93b5d73f24f75afbcb7660706db8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.241001 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb595013e809480a814cb9c7aed856e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:15.241219 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93c93b5d73f24f75afbcb7660706db8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.241456 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.241468 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f14f06ba7d0f44418693e36f6bc6d888 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
I20250110 02:51:15.242111 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_306f9f127f75420882d16d81923bf9f5" } modify_external_catalogs: true
I20250110 02:51:15.242252 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_306f9f127f75420882d16d81923bf9f5" } modify_external_catalogs: true
I20250110 02:51:15.243320 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_61e4e1bf02224dce873e1aa1bf273c54" } new_table_name: "default.Table_9064b36ee7684e2682fc53f68bcf1f47" modify_external_catalogs: true
I20250110 02:51:15.243818 1179 tablet_replica.cc:331] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.243990 1179 raft_consensus.cc:2238] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.244155 1179 raft_consensus.cc:2267] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.244357 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e5f306c95c94407da3a3eff0c31e33a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.244740 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5f306c95c94407da3a3eff0c31e33a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.245355 1184 raft_consensus.cc:357] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.245535 1184 raft_consensus.cc:383] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.229859 1188 raft_consensus.cc:357] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.246064 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467f87955c4a4d27a6f0483aa109b922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.246016 1188 raft_consensus.cc:383] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.246901 1188 raft_consensus.cc:738] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.248011 1179 ts_tablet_manager.cc:1905] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.248003 1545 raft_consensus.cc:491] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.248116 1545 raft_consensus.cc:513] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.249248 1545 leader_election.cc:290] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.249264 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.249449 1061 raft_consensus.cc:2463] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:15.249886 853 leader_election.cc:304] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:15.250135 1545 raft_consensus.cc:2798] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:15.250236 1545 raft_consensus.cc:491] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:15.250316 1545 raft_consensus.cc:3054] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.252839 1545 raft_consensus.cc:513] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.253345 1545 leader_election.cc:290] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.254513 885 tablet_service.cc:1467] Processing CreateTablet for tablet ae3033595f824affb2d300c3964e1e8e (DEFAULT_TABLE table=default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.254897 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3033595f824affb2d300c3964e1e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.255357 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:15.255529 1061 raft_consensus.cc:3054] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.258221 1061 raft_consensus.cc:2463] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:15.258644 881 tablet_service.cc:1467] Processing CreateTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.259089 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e554d00d05b4fa7a14c9b6159f9d726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.262665 892 tablet_service.cc:1467] Processing CreateTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.262751 853 leader_election.cc:304] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:15.262982 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b90cc15dbf47d4b4dbaf12d588abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.266539 1181 consensus_queue.cc:260] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.268976 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.269371 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e554d00d05b4fa7a14c9b6159f9d726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.270836 775 raft_consensus.cc:2463] T 1cc0532c94eb401e9d299e79556b4cf1 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:15.274628 1181 ts_tablet_manager.cc:1428] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20250110 02:51:15.275933 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.276257 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b90cc15dbf47d4b4dbaf12d588abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.278283 996 leader_election.cc:304] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:15.278726 1386 raft_consensus.cc:2798] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:15.280586 1386 raft_consensus.cc:695] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:15.280826 1386 consensus_queue.cc:237] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.274947 1184 raft_consensus.cc:738] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.282465 775 raft_consensus.cc:3054] T 8162ceb17c714614b34beb33a7633ff0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.282562 1184 consensus_queue.cc:260] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.287123 1545 raft_consensus.cc:2798] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:15.287128 756 tablet_service.cc:1467] Processing CreateTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 (DEFAULT_TABLE table=default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.288686 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b90cc15dbf47d4b4dbaf12d588abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.288846 754 tablet_service.cc:1467] Processing CreateTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (DEFAULT_TABLE table=default.Table_db04ddcfa7864a5a82f05df887a858aa [id=953ce2773b0041909b2e1f7767f361bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.289619 749 tablet_service.cc:1467] Processing CreateTablet for tablet ae3033595f824affb2d300c3964e1e8e (DEFAULT_TABLE table=default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.290161 1184 ts_tablet_manager.cc:1428] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
W20250110 02:51:15.292501 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 467f87955c4a4d27a6f0483aa109b922 consensus state has no leader
I20250110 02:51:15.294224 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.294153 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_69e74c02a7094d698547edc4e145bdd4" } new_table_name: "default.Table_618cae6951b649328aab854b7efa2a14" modify_external_catalogs: true
I20250110 02:51:15.294508 1178 tablet_replica.cc:331] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:15.294481 518 master_service.cc:946] ReplaceTablet: received request to replace tablet 76833ce515cf4ee9bc8fff0f626af220 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.294684 1178 raft_consensus.cc:2238] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.294885 1178 raft_consensus.cc:2267] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.295414 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e554d00d05b4fa7a14c9b6159f9d726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.295575 1178 ts_tablet_manager.cc:1905] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.288115 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:15.296566 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c0edfa33e30042edad853217aa761bd5
I20250110 02:51:15.296505 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e7baf08453f44f56a442f1d355c8a32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.297003 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7baf08453f44f56a442f1d355c8a32c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.297212 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5430a61e23f84d95b159e6d406f72eef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.297652 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5430a61e23f84d95b159e6d406f72eef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.298391 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6f1e9a903a2e4ecaba8ded1a3f775428"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.298738 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 consensus state has no leader
W20250110 02:51:15.298918 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f1e9a903a2e4ecaba8ded1a3f775428 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.298947 897 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edfa33e30042edad853217aa761bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.299053 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_fe466b4cc96c4d50a4ed42daae488818" } new_table_name: "default.Table_8f356d58321749be9aed0b015ff38d71" modify_external_catalogs: true
I20250110 02:51:15.300927 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e25f9643ac0476a90c93fb9aeb921a3 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.301309 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.302809 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7431ea53e3704aebb4d79e0a0c4045f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.303210 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7431ea53e3704aebb4d79e0a0c4045f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.304296 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_36aaf3ef6c99465eb16fb6565a677486" } modify_external_catalogs: true
I20250110 02:51:15.288712 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86be26d1b544c08b231f20085d3159" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:15.304435 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_36aaf3ef6c99465eb16fb6565a677486" } modify_external_catalogs: true
I20250110 02:51:15.304564 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3033595f824affb2d300c3964e1e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.305455 773 raft_consensus.cc:3054] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.307112 917 raft_consensus.cc:3054] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.312214 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.312688 1184 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.314231 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edfa33e30042edad853217aa761bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.314417 1599 consensus_queue.cc:1035] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:15.314606 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edfa33e30042edad853217aa761bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.317279 773 raft_consensus.cc:2463] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:15.331624 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 467f87955c4a4d27a6f0483aa109b922 (DEFAULT_TABLE table=default.Table_61e4e1bf02224dce873e1aa1bf273c54 [id=b58434c397244de09dd22be49baeeb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.332141 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467f87955c4a4d27a6f0483aa109b922. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.336050 1599 consensus_queue.cc:1035] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:15.339223 1599 consensus_queue.cc:1035] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:15.340289 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:15.247767 1188 consensus_queue.cc:260] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.340698 518 catalog_manager.cc:6596] ReplaceTablet: tablet 76833ce515cf4ee9bc8fff0f626af220 deleted and replaced by tablet b74beb77505f472fb6b9e4a30bfe6582
I20250110 02:51:15.340961 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.340983 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a1daaf2964134078b6e9ac86b7672a37" } modify_external_catalogs: true
I20250110 02:51:15.341145 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_a1daaf2964134078b6e9ac86b7672a37" } modify_external_catalogs: true
I20250110 02:51:15.342181 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3375784ee4514b079aaf4ef21656d9ec" } modify_external_catalogs: true
I20250110 02:51:15.343000 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3375784ee4514b079aaf4ef21656d9ec" } modify_external_catalogs: true
I20250110 02:51:15.343117 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ad6ed08b6a0e47599bd592fbaf2c5090" } modify_external_catalogs: true
I20250110 02:51:15.343219 1178 ts_tablet_manager.cc:1918] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.343271 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ad6ed08b6a0e47599bd592fbaf2c5090" } modify_external_catalogs: true
I20250110 02:51:15.343340 1178 log.cc:1198] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c65704c9d486444ebe8e4ffd4580f395
W20250110 02:51:15.344862 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac5dab241cd74f329a57e6f19e95a7b6 has no consensus state
I20250110 02:51:15.345103 1184 tablet_bootstrap.cc:654] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.347000 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.348062 1179 ts_tablet_manager.cc:1918] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.351491 1178 ts_tablet_manager.cc:1939] T c65704c9d486444ebe8e4ffd4580f395 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:15.353185 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c65704c9d486444ebe8e4ffd4580f395 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:15.354143 1106 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:15.354547 1178 tablet_replica.cc:331] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:15.354786 1178 raft_consensus.cc:2238] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.355005 1178 raft_consensus.cc:2267] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.355453 1588 raft_consensus.cc:491] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.355613 1178 ts_tablet_manager.cc:1905] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.355924 1588 raft_consensus.cc:513] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.356359 918 raft_consensus.cc:3054] T d5f56778a0244378ad8499a4ceef3510 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.356580 1588 leader_election.cc:290] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.357213 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.357331 1179 log.cc:1198] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e744cdc1fcb94d46b48b684763ecf1bf
W20250110 02:51:15.357712 711 leader_election.cc:343] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.361347 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.361970 1599 consensus_queue.cc:1035] T d5f56778a0244378ad8499a4ceef3510 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:15.362447 709 leader_election.cc:343] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.364699 709 leader_election.cc:304] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.365100 1588 raft_consensus.cc:2743] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.365444 1181 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:15.367224 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abbd43638a2c47dcba4f6f90e8676f42 consensus state has no leader
I20250110 02:51:15.369282 1188 ts_tablet_manager.cc:1428] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.154s user 0.003s sys 0.001s
I20250110 02:51:15.370350 1188 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:15.370702 1179 ts_tablet_manager.cc:1939] T e744cdc1fcb94d46b48b684763ecf1bf P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.371188 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e744cdc1fcb94d46b48b684763ecf1bf (table default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]) successfully deleted
I20250110 02:51:15.371256 1179 tablet_replica.cc:331] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.371515 1179 raft_consensus.cc:2238] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.371701 1179 raft_consensus.cc:2267] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.372059 1178 ts_tablet_manager.cc:1918] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:15.372397 1179 ts_tablet_manager.cc:1905] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.373025 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.378144 1178 log.cc:1198] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c0edfa33e30042edad853217aa761bd5
I20250110 02:51:15.378757 1178 ts_tablet_manager.cc:1939] T c0edfa33e30042edad853217aa761bd5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:15.379801 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]) successfully deleted
W20250110 02:51:15.380116 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76833ce515cf4ee9bc8fff0f626af220 already in progress: creating tablet
I20250110 02:51:15.380234 1181 tablet_bootstrap.cc:654] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.382367 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ae3033595f824affb2d300c3964e1e8e (DEFAULT_TABLE table=default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.383025 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3033595f824affb2d300c3964e1e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.384490 1180 ts_tablet_manager.cc:1918] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.389621 1179 ts_tablet_manager.cc:1918] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.390890 1181 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.392648 1181 ts_tablet_manager.cc:1397] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250110 02:51:15.391489 1180 log.cc:1198] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/64c77ec8c7724d5fb408a8bfa53811eb
I20250110 02:51:15.394793 1180 ts_tablet_manager.cc:1939] T 64c77ec8c7724d5fb408a8bfa53811eb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.395484 1179 log.cc:1198] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/46a4745356e3423f9d4892d242da1c48
I20250110 02:51:15.396080 1179 ts_tablet_manager.cc:1939] T 46a4745356e3423f9d4892d242da1c48 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.397379 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 64c77ec8c7724d5fb408a8bfa53811eb (table default.Table_e676f1148b5f4988a2be33981eec57c3 [id=d7bbf5b39b224708974b5f6da6429fe7]) successfully deleted
I20250110 02:51:15.398713 1180 tablet_replica.cc:331] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.400272 1180 raft_consensus.cc:2238] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.404214 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 46a4745356e3423f9d4892d242da1c48 (table default.Table_123de8ace5ed47f7850040c5de4fcbc7 [id=3aad422d584f40df9d9241586adbc66f]) successfully deleted
I20250110 02:51:15.405046 1180 raft_consensus.cc:2267] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.405344 1184 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.405529 1184 ts_tablet_manager.cc:1397] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.007s
W20250110 02:51:15.406131 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d970e2fe72b4727968ca605b3e772cc already in progress: creating tablet
I20250110 02:51:15.406339 1184 raft_consensus.cc:357] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.406594 1184 raft_consensus.cc:383] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.406689 1184 raft_consensus.cc:738] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.394270 1181 raft_consensus.cc:357] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.406939 1181 raft_consensus.cc:383] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.406970 1184 consensus_queue.cc:260] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.407474 1184 ts_tablet_manager.cc:1428] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:15.407783 1184 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.407629 1181 raft_consensus.cc:738] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:15.408895 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08705d260c0a462b94ba6b0dc5121123 already in progress: creating tablet
I20250110 02:51:15.410773 1545 raft_consensus.cc:695] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:15.410951 1467 raft_consensus.cc:491] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.411065 1467 raft_consensus.cc:513] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.411084 1181 consensus_queue.cc:260] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.411070 1545 consensus_queue.cc:237] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:15.411708 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:15.411731 1467 leader_election.cc:290] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:15.412714 996 leader_election.cc:343] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 1dfc062b5fcb42c3bce6dfaed54b3869
I20250110 02:51:15.416428 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfc062b5fcb42c3bce6dfaed54b3869" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.417284 1180 ts_tablet_manager.cc:1905] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.372886 1188 tablet_bootstrap.cc:654] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:15.418768 994 leader_election.cc:343] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.425238 994 leader_election.cc:304] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:15.418944 1599 raft_consensus.cc:491] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.425576 1599 raft_consensus.cc:513] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.426155 1599 leader_election.cc:290] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.428872 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet e015a41a03f84c91a45e46e4b7b5e7a9
W20250110 02:51:15.429087 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bd2283e7ed7456dae466a8582587eb7 has no consensus state
I20250110 02:51:15.429524 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.429584 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:15.430043 525 catalog_manager.cc:6596] ReplaceTablet: tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 deleted and replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62
I20250110 02:51:15.430284 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.430963 517 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2dd756ca7d0c4f99ba072f1f40db22f3
I20250110 02:51:15.431293 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c5601ca24b8f428e8a51cbeec3664901
I20250110 02:51:15.431854 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:15.433041 1184 tablet_bootstrap.cc:654] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.433890 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.434299 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.434968 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.435024 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.434963 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:15.435621 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
I20250110 02:51:15.435729 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:15.436198 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 already in progress: creating tablet
I20250110 02:51:15.436422 749 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.436673 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
I20250110 02:51:15.437251 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:15.437319 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65285bce60fd4015b085deb7442eea6e already in progress: creating tablet
I20250110 02:51:15.438108 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.444478 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:15.444610 510 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ac5dab241cd74f329a57e6f19e95a7b6 has no consensus state
I20250110 02:51:15.447075 1181 ts_tablet_manager.cc:1428] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
W20250110 02:51:15.452946 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 95b14d7592024a34baf7756d92e546d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95b14d7592024a34baf7756d92e546d9 already in progress: creating tablet
I20250110 02:51:15.455031 1181 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.457468 1179 tablet_replica.cc:331] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:15.457720 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet da8f247c862e4cb68452e230857ee6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da8f247c862e4cb68452e230857ee6b0 already in progress: creating tablet
I20250110 02:51:15.459412 1179 raft_consensus.cc:2238] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.459692 1179 raft_consensus.cc:2267] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.461115 1179 ts_tablet_manager.cc:1905] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.478586 1633 raft_consensus.cc:2743] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.485644 524 master_service.cc:946] ReplaceTablet: received request to replace tablet b2163c343a694f5ca04ac9f7610d25e3 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:15.479163 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 60411.3us
I20250110 02:51:15.456835 1181 tablet_bootstrap.cc:654] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.486199 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514704cd935b44e8b760221224332509" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.486475 916 raft_consensus.cc:2463] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:15.487046 1599 raft_consensus.cc:491] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.487232 994 leader_election.cc:304] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:15.487201 1599 raft_consensus.cc:513] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.487740 1599 leader_election.cc:290] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:15.488214 1599 raft_consensus.cc:2798] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:15.488282 1599 raft_consensus.cc:491] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:15.488353 1599 raft_consensus.cc:3054] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:15.490971 1188 tablet_bootstrap.cc:492] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:15.491197 1188 ts_tablet_manager.cc:1397] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.008s
I20250110 02:51:15.491963 1599 raft_consensus.cc:513] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.492498 1599 leader_election.cc:290] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.492746 1188 raft_consensus.cc:357] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.493008 1188 raft_consensus.cc:383] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.499675 1633 consensus_queue.cc:1035] T 8162ceb17c714614b34beb33a7633ff0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:15.499924 994 leader_election.cc:343] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:15.482949 1180 ts_tablet_manager.cc:1918] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 02:51:15.485541 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 63146.1us
I20250110 02:51:15.505160 1180 log.cc:1198] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/423c820b81b845719d59a376c18e175a
I20250110 02:51:15.505600 1180 ts_tablet_manager.cc:1939] T 423c820b81b845719d59a376c18e175a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.506951 517 master_service.cc:946] ReplaceTablet: received request to replace tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.507337 1181 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.508029 1181 ts_tablet_manager.cc:1397] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250110 02:51:15.509231 1181 raft_consensus.cc:357] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.509606 1181 raft_consensus.cc:383] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.509734 1181 raft_consensus.cc:738] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.510564 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_5430a61e23f84d95b159e6d406f72eef" } new_table_name: "default.Table_0317f9a01941445699da40e583bbe394" modify_external_catalogs: true
I20250110 02:51:15.514868 519 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ee284925f5344933a090d3f345f45b53" } modify_external_catalogs: true
I20250110 02:51:15.515084 519 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_ee284925f5344933a090d3f345f45b53" } modify_external_catalogs: true
I20250110 02:51:15.515381 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c2fd603cae0f4558877bff141025cd02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.516175 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_bb1c63b5d7ee453a999514a777e71496"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.516865 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3e713a5d93834ea3be890e774e266d83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.519189 1181 consensus_queue.cc:260] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:15.521871 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04a4e7142e83493b98f71487df3f304d has no consensus state
I20250110 02:51:15.522230 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b0b62b5867624cd1b77150b7b3d2090d" } new_table_name: "default.Table_94cf6a20190647dd87d3616b069b0243" modify_external_catalogs: true
W20250110 02:51:15.524569 996 leader_election.cc:343] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 514704cd935b44e8b760221224332509
I20250110 02:51:15.495290 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514704cd935b44e8b760221224332509" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:15.526432 915 raft_consensus.cc:3054] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:15.527343 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7397e41f7374e9fb1621dbcf30af703 consensus state has no leader
I20250110 02:51:15.527688 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_61c4e10750aa4566916853b2c35ed9fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.529182 915 raft_consensus.cc:2463] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:15.529562 1620 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8bc934cc708045e9858ee8f8e64660cd)
I20250110 02:51:15.529703 994 leader_election.cc:304] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:15.529846 1620 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:15.530225 1636 raft_consensus.cc:2798] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:15.530405 1636 raft_consensus.cc:695] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:15.531149 1636 consensus_queue.cc:237] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.533996 1181 ts_tablet_manager.cc:1428] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250110 02:51:15.534035 1577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc934cc708045e9858ee8f8e64660cd. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:15.534283 1577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:15.536451 1481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39276 (request call id 0) took 1107 ms. Trace:
W20250110 02:51:15.541132 996 leader_election.cc:343] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:15.541669 996 leader_election.cc:304] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:15.542606 1635 raft_consensus.cc:2743] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:15.555259 996 consensus_peers.cc:487] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:15.567562 1620 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
W20250110 02:51:15.574661 996 leader_election.cc:343] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 514704cd935b44e8b760221224332509
I20250110 02:51:15.576754 1181 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.581032 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 423c820b81b845719d59a376c18e175a (table default.Table_444895c40f0f4c2fb8377fa6ffa6a4c5 [id=ee252f8d627a41db8db58fa208b8f1d4]) successfully deleted
W20250110 02:51:15.583959 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e713a5d93834ea3be890e774e266d83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:15.585042 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb1c63b5d7ee453a999514a777e71496 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:15.587061 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2fd603cae0f4558877bff141025cd02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:15.592312 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61c4e10750aa4566916853b2c35ed9fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.596714 1184 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.597147 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_98b0cdca19ab460ca64c35faf958d9bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.598752 1188 raft_consensus.cc:738] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.599154 1188 consensus_queue.cc:260] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.600037 1188 ts_tablet_manager.cc:1428] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.109s user 0.002s sys 0.000s
I20250110 02:51:15.600415 1188 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:15.602495 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b0cdca19ab460ca64c35faf958d9bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.599074 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a2d5d1bf3c504747b630f2dd924d83f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.605441 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2d5d1bf3c504747b630f2dd924d83f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.610016 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_6155a05a8ab843e8a0ad9f115a26d4e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.610757 1184 ts_tablet_manager.cc:1397] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.010s
W20250110 02:51:15.611447 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.601554 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_febf1d5b7a274c5fb59e10a180b4dd59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.616895 1184 raft_consensus.cc:357] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.617180 1184 raft_consensus.cc:383] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.617276 1184 raft_consensus.cc:738] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.617579 1184 consensus_queue.cc:260] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.618388 1184 ts_tablet_manager.cc:1428] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250110 02:51:15.617908 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_febf1d5b7a274c5fb59e10a180b4dd59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.623739 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_526de8fc3ad6495ea9ec7e55e2accd58" } new_table_name: "default.Table_9ca42c50470d4cebaf832606cb958d23" modify_external_catalogs: true
I20250110 02:51:15.628861 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_34eb9282d67443a2849ec525cdaa9e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:15.629638 1588 raft_consensus.cc:491] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.629807 1588 raft_consensus.cc:513] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.630432 1588 leader_election.cc:290] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.632023 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ae1c19d2dc4778a1cebddbd505f907" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:15.632946 711 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.634574 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ae1c19d2dc4778a1cebddbd505f907" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:15.635133 709 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.635253 709 leader_election.cc:304] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.635396 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_88abc71d734244088bac8733189eae55" } new_table_name: "default.Table_0c2624c5c6b646e1969b7e0abedf82e0" modify_external_catalogs: true
I20250110 02:51:15.635749 1588 raft_consensus.cc:2743] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.641433 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1dc0c1e6f84c4bff8bc4c249ac502084" } new_table_name: "default.Table_8c793fc0a0294bcf925860d30901b7d0" modify_external_catalogs: true
W20250110 02:51:15.648833 994 consensus_peers.cc:487] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0edfa33e30042edad853217aa761bd5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:15.649223 1188 tablet_bootstrap.cc:654] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.581574 1180 tablet_replica.cc:331] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.650728 1180 raft_consensus.cc:2238] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:15.651028 1180 pending_rounds.cc:62] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:15.651120 1180 pending_rounds.cc:69] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611737124990976 op_type: NO_OP noop_request { }
I20250110 02:51:15.651262 1180 raft_consensus.cc:2883] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:15.651388 1180 raft_consensus.cc:2267] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:15.629864 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34eb9282d67443a2849ec525cdaa9e7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.655257 1651 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250110 02:51:15.656653 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19e4af23720048fe98b3231ba402d45c consensus state has no leader
I20250110 02:51:15.660167 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e04ee12d5f244793a7a44387a6243c1d" } modify_external_catalogs: true
I20250110 02:51:15.660446 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e04ee12d5f244793a7a44387a6243c1d" } modify_external_catalogs: true
I20250110 02:51:15.666636 1181 tablet_bootstrap.cc:654] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.677079 1181 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.677318 1181 ts_tablet_manager.cc:1397] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.000s
I20250110 02:51:15.678134 1181 raft_consensus.cc:357] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.678488 1181 raft_consensus.cc:383] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.678617 1181 raft_consensus.cc:738] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.678885 1181 consensus_queue.cc:260] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.679122 1179 ts_tablet_manager.cc:1918] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:15.680297 1179 log.cc:1198] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c0edfa33e30042edad853217aa761bd5
I20250110 02:51:15.680717 1179 ts_tablet_manager.cc:1939] T c0edfa33e30042edad853217aa761bd5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.684376 1457 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" instance_seqno: 1736477468177459) as {username='slave'} at 127.0.104.3:37843; Asking this server to re-register.
I20250110 02:51:15.684870 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]) successfully deleted
W20250110 02:51:15.685156 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76833ce515cf4ee9bc8fff0f626af220 already in progress: creating tablet
W20250110 02:51:15.685364 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
W20250110 02:51:15.685534 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 already in progress: creating tablet
W20250110 02:51:15.685663 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
I20250110 02:51:15.685752 1456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "85f9597077934f1a9814d284ed06d13d" instance_seqno: 1736477467615051) as {username='slave'} at 127.0.104.2:47877; Asking this server to re-register.
W20250110 02:51:15.685825 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65285bce60fd4015b085deb7442eea6e already in progress: creating tablet
I20250110 02:51:15.686453 963 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:15.688416 1457 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "98336b684b41407da8a1949e30cd82e0" instance_seqno: 1736477467131857) as {username='slave'} at 127.0.104.1:57009; Asking this server to re-register.
I20250110 02:51:15.689112 1106 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:15.689121 1456 ts_manager.cc:194] Registered new tserver with Master: 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:15.690586 822 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:15.690742 1456 ts_manager.cc:194] Registered new tserver with Master: 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.691640 1181 ts_tablet_manager.cc:1428] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250110 02:51:15.692026 1181 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.693320 1456 ts_manager.cc:194] Registered new tserver with Master: 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.695695 1184 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.697219 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_82157dfbcfe64e759c6847722284ee3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.697554 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 871180863899419595cd632e7a4cf0dc consensus state has no leader
I20250110 02:51:15.697643 1184 tablet_bootstrap.cc:654] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:15.697806 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82157dfbcfe64e759c6847722284ee3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.697885 523 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3c02731f843a4cc3b13b54dee5b9e7f7" } modify_external_catalogs: true
I20250110 02:51:15.698063 523 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3c02731f843a4cc3b13b54dee5b9e7f7" } modify_external_catalogs: true
I20250110 02:51:15.702801 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:15.702989 521 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:15.704238 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0e2df09ed397432f8697c6baf59aaa53" } modify_external_catalogs: true
I20250110 02:51:15.704360 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_0e2df09ed397432f8697c6baf59aaa53" } modify_external_catalogs: true
I20250110 02:51:15.705504 526 catalog_manager.cc:5526] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:15.708031 522 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:15.712795 1180 ts_tablet_manager.cc:1905] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.713264 881 tablet_service.cc:1467] Processing CreateTablet for tablet 04a4e7142e83493b98f71487df3f304d (DEFAULT_TABLE table=default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.713620 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a4e7142e83493b98f71487df3f304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.714609 885 tablet_service.cc:1467] Processing CreateTablet for tablet e27f7ffc50494854ba34e1fbce9bee1b (DEFAULT_TABLE table=default.Table_e7baf08453f44f56a442f1d355c8a32c [id=41e43455ee7c45a6960721c33a18b4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.714934 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27f7ffc50494854ba34e1fbce9bee1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.715185 1588 raft_consensus.cc:491] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.715358 1588 raft_consensus.cc:513] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.715967 1588 leader_election.cc:290] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.716694 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb5412b78104d8085878c58b191ab27" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:15.717442 711 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.717442 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb5412b78104d8085878c58b191ab27" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.717790 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 461 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
W20250110 02:51:15.717996 709 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.718127 709 leader_election.cc:304] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.718585 1588 raft_consensus.cc:2743] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.723639 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_564f2d190f904222b9f705210a88ac5c" } modify_external_catalogs: true
I20250110 02:51:15.723803 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_564f2d190f904222b9f705210a88ac5c" } modify_external_catalogs: true
W20250110 02:51:15.723959 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c6de957fd14af983fc1a3cd8629226 consensus state has no leader
I20250110 02:51:15.726665 749 tablet_service.cc:1467] Processing CreateTablet for tablet 04a4e7142e83493b98f71487df3f304d (DEFAULT_TABLE table=default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.727099 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a4e7142e83493b98f71487df3f304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.727051 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 04a4e7142e83493b98f71487df3f304d (DEFAULT_TABLE table=default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.727448 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a4e7142e83493b98f71487df3f304d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.729439 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:15.729729 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:15.730504 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.730639 754 tablet_service.cc:1467] Processing CreateTablet for tablet e27f7ffc50494854ba34e1fbce9bee1b (DEFAULT_TABLE table=default.Table_e7baf08453f44f56a442f1d355c8a32c [id=41e43455ee7c45a6960721c33a18b4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.730984 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27f7ffc50494854ba34e1fbce9bee1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.730952 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e27f7ffc50494854ba34e1fbce9bee1b (DEFAULT_TABLE table=default.Table_e7baf08453f44f56a442f1d355c8a32c [id=41e43455ee7c45a6960721c33a18b4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.731226 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27f7ffc50494854ba34e1fbce9bee1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.734664 753 tablet_service.cc:1467] Processing CreateTablet for tablet 13683fbe9d654575b28108de8b77f943 (DEFAULT_TABLE table=default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 [id=2fe05e5dfd244de7838c0642fa521be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.734664 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 13683fbe9d654575b28108de8b77f943 (DEFAULT_TABLE table=default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 [id=2fe05e5dfd244de7838c0642fa521be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.735024 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13683fbe9d654575b28108de8b77f943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.735024 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13683fbe9d654575b28108de8b77f943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.737422 520 master_service.cc:946] ReplaceTablet: received request to replace tablet d20177229f7b4f11875e0007afb1dd00 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.737844 520 master_service.cc:946] ReplaceTablet: received request to replace tablet d761ef0d1e5542b1979c29bb2d579489 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.738648 1039 tablet_service.cc:1467] Processing CreateTablet for tablet e2ab2376576448deb87f607b3e5a4f1c (DEFAULT_TABLE table=default.Table_ce235042d71840c0a05ae04e253c9517 [id=5ba8eda7053b45e2b27844e97e5c513a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.738662 756 tablet_service.cc:1467] Processing CreateTablet for tablet e2ab2376576448deb87f607b3e5a4f1c (DEFAULT_TABLE table=default.Table_ce235042d71840c0a05ae04e253c9517 [id=5ba8eda7053b45e2b27844e97e5c513a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.738953 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ab2376576448deb87f607b3e5a4f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.738978 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ab2376576448deb87f607b3e5a4f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.742637 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 9395aa9b39d04ac9b6cecf106bc91499 (DEFAULT_TABLE table=default.Table_adb6100fd2ea489590831f0508f53e3e [id=811a01d9de1a4185a876251bd8d10512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.742632 745 tablet_service.cc:1467] Processing CreateTablet for tablet 9395aa9b39d04ac9b6cecf106bc91499 (DEFAULT_TABLE table=default.Table_adb6100fd2ea489590831f0508f53e3e [id=811a01d9de1a4185a876251bd8d10512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.742954 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9395aa9b39d04ac9b6cecf106bc91499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.742920 1180 ts_tablet_manager.cc:1918] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:15.743085 1180 log.cc:1198] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/70af6ec40c7a4442b91680d8125c2d48
I20250110 02:51:15.743244 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a (DEFAULT_TABLE table=default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.743618 1180 ts_tablet_manager.cc:1939] T 70af6ec40c7a4442b91680d8125c2d48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.742970 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9395aa9b39d04ac9b6cecf106bc91499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.744602 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 353) took 1063 ms. Trace:
I20250110 02:51:15.744736 1180 rpcz_store.cc:276] 0110 02:51:14.681028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:14.681069 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:15.581439 (+900370us) ts_tablet_manager.cc:1145] Deleting tablet 70af6ec40c7a4442b91680d8125c2d48
0110 02:51:15.581449 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:15.716240 (+134791us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:15.742770 (+ 26530us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:15.744546 (+ 1776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":900179}
I20250110 02:51:15.746614 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a (DEFAULT_TABLE table=default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.746949 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.746984 1184 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.747138 1184 ts_tablet_manager.cc:1397] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
I20250110 02:51:15.747931 1184 raft_consensus.cc:357] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.748190 1184 raft_consensus.cc:383] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.748268 1184 raft_consensus.cc:738] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.750296 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.750525 1179 tablet_replica.cc:331] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.750758 1179 raft_consensus.cc:2238] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.750908 1179 raft_consensus.cc:2267] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.751628 1179 ts_tablet_manager.cc:1905] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.753453 1184 consensus_queue.cc:260] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.754115 1180 tablet_replica.cc:331] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.754328 1180 raft_consensus.cc:2238] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.755769 1180 raft_consensus.cc:2267] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.755826 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 70af6ec40c7a4442b91680d8125c2d48 (table default.Table_6365c4bf56b44696a42c21a809c8210c [id=f01044e5baf54b17a9a56cc124bd6605]) successfully deleted
I20250110 02:51:15.756309 1180 ts_tablet_manager.cc:1905] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.758425 1184 ts_tablet_manager.cc:1428] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250110 02:51:15.762148 1184 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.763370 1179 ts_tablet_manager.cc:1918] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.763504 1179 log.cc:1198] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:15.763871 1179 ts_tablet_manager.cc:1939] T 65ae1c19d2dc4778a1cebddbd505f907 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:15.764886 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 65ae1c19d2dc4778a1cebddbd505f907 (table default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]) successfully deleted
I20250110 02:51:15.766462 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:15.766928 517 catalog_manager.cc:6596] ReplaceTablet: tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa deleted and replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a
I20250110 02:51:15.767132 517 master_service.cc:946] ReplaceTablet: received request to replace tablet c5601ca24b8f428e8a51cbeec3664901 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:15.767696 517 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_6f1e9a903a2e4ecaba8ded1a3f775428" } new_table_name: "default.Table_56ef844e399040a3ac6279fc5d2ed20b" modify_external_catalogs: true
I20250110 02:51:15.769125 524 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:15.769488 524 catalog_manager.cc:6596] ReplaceTablet: tablet b2163c343a694f5ca04ac9f7610d25e3 deleted and replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f
I20250110 02:51:15.769721 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4a33b1b80f7f40058628dfb357187113" } new_table_name: "default.Table_d62f105412d049af86f21ec909ac07b5" modify_external_catalogs: true
I20250110 02:51:15.769862 756 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.770314 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.770534 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
I20250110 02:51:15.770748 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:15.770882 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2163c343a694f5ca04ac9f7610d25e3 already in progress: creating tablet
I20250110 02:51:15.772115 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.773320 1184 tablet_bootstrap.cc:654] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.775182 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:15.775362 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]). Expected version 1 got 0
I20250110 02:51:15.775693 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:15.777386 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.767774 518 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 3 got 0
I20250110 02:51:15.779417 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 213939cf13354ce380fb7cda5b38676f
I20250110 02:51:15.780182 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.781970 1588 raft_consensus.cc:491] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.782054 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.782164 1588 raft_consensus.cc:513] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:15.782564 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because the tablet was not found. No further retry: Not found: Tablet not found: 213939cf13354ce380fb7cda5b38676f
I20250110 02:51:15.783216 1588 leader_election.cc:290] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.784119 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:15.784173 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.784679 519 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet a7d4f03a8c66446ea7c6d178cc7ef80a
W20250110 02:51:15.784665 711 leader_election.cc:343] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.785153 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.785368 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet d761ef0d1e5542b1979c29bb2d579489
W20250110 02:51:15.785753 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because the tablet was not found. No further retry: Not found: Tablet not found: 5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:15.785782 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.785827 520 catalog_manager.cc:6596] ReplaceTablet: tablet d761ef0d1e5542b1979c29bb2d579489 deleted and replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe
I20250110 02:51:15.786327 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.786175 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because the tablet was not found. No further retry: Not found: Tablet not found: a7d4f03a8c66446ea7c6d178cc7ef80a
I20250110 02:51:15.786541 519 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 3 got 0
I20250110 02:51:15.786648 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.787155 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.787357 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.787979 523 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 04e05a7554a14805bdcbd1824a5bd9d5
W20250110 02:51:15.787992 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
I20250110 02:51:15.788583 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.788669 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.789599 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e05a7554a14805bdcbd1824a5bd9d5 already in progress: creating tablet
I20250110 02:51:15.790617 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:15.791308 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:15.791392 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:15.791760 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 already in progress: creating tablet
I20250110 02:51:15.794898 1180 ts_tablet_manager.cc:1918] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.795023 1180 log.cc:1198] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:15.795333 1180 ts_tablet_manager.cc:1939] T ad9f920cde954f31acebc6d0b6965e58 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.795995 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 354) took 1114 ms. Trace:
I20250110 02:51:15.796080 1180 rpcz_store.cc:276] 0110 02:51:14.681445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:14.681483 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:15.754053 (+1072570us) ts_tablet_manager.cc:1145] Deleting tablet ad9f920cde954f31acebc6d0b6965e58
0110 02:51:15.754060 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:15.782166 (+ 28106us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:15.794804 (+ 12638us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:15.795950 (+ 1146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072406}
I20250110 02:51:15.797901 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9b1e53691288443193e1896ddc095952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.798416 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b1e53691288443193e1896ddc095952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.799916 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_375ea34c1b6d471ca632286504f259f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.800293 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_375ea34c1b6d471ca632286504f259f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.802732 1184 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.802889 1184 ts_tablet_manager.cc:1397] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.007s
I20250110 02:51:15.803607 1184 raft_consensus.cc:357] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.803906 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_37fa37bb314c4af39a4bb95697714acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.804244 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37fa37bb314c4af39a4bb95697714acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.804313 1184 raft_consensus.cc:383] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.804412 1184 raft_consensus.cc:738] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.804622 1184 consensus_queue.cc:260] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.805124 1184 ts_tablet_manager.cc:1428] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:15.805428 1180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55354 (request call id 356) took 1122 ms. Trace:
I20250110 02:51:15.805527 1180 rpcz_store.cc:276] 0110 02:51:14.682839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:14.682897 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:15.805268 (+1122371us) ts_tablet_manager.cc:1145] Deleting tablet 28340f0f9a2043a199c74f13755eb702
0110 02:51:15.805275 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:15.805389 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122149}
I20250110 02:51:15.806190 1180 tablet_replica.cc:331] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.806357 1180 raft_consensus.cc:2238] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.806550 1180 raft_consensus.cc:2267] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.807109 1180 ts_tablet_manager.cc:1905] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.809877 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c65704c9d486444ebe8e4ffd4580f395 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c0d22fd75fb474ebf07b624499a6f7b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.815057 756 tablet_service.cc:1467] Processing CreateTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 (DEFAULT_TABLE table=default.Table_93c93b5d73f24f75afbcb7660706db8a [id=5abe0f9c7bb9402c941464132d4ca4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.815413 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eadb829d657346eabcb7d2aa34b33fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.815696 1184 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.817883 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
I20250110 02:51:15.819487 897 tablet_service.cc:1467] Processing CreateTablet for tablet e2ab2376576448deb87f607b3e5a4f1c (DEFAULT_TABLE table=default.Table_ce235042d71840c0a05ae04e253c9517 [id=5ba8eda7053b45e2b27844e97e5c513a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.819886 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ab2376576448deb87f607b3e5a4f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.821535 1029 tablet_service.cc:1467] Processing CreateTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 (DEFAULT_TABLE table=default.Table_93c93b5d73f24f75afbcb7660706db8a [id=5abe0f9c7bb9402c941464132d4ca4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.821997 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eadb829d657346eabcb7d2aa34b33fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.822064 1180 ts_tablet_manager.cc:1918] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.822177 1184 tablet_bootstrap.cc:654] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.822186 1180 log.cc:1198] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/4e2c01af512f460daa98b19444f72172
I20250110 02:51:15.822566 1180 ts_tablet_manager.cc:1939] T 4e2c01af512f460daa98b19444f72172 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.825297 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
W20250110 02:51:15.826779 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c65704c9d486444ebe8e4ffd4580f395 because the tablet was not found. No further retry: Not found: Tablet not found: c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:15.827385 1180 tablet_replica.cc:331] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.827594 1180 raft_consensus.cc:2238] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:15.827842 1180 pending_rounds.cc:62] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:15.827925 1180 pending_rounds.cc:69] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611739778392064 op_type: NO_OP noop_request { }
I20250110 02:51:15.828042 1180 raft_consensus.cc:2883] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:15.828725 1180 raft_consensus.cc:2267] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.828900 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9395aa9b39d04ac9b6cecf106bc91499 (DEFAULT_TABLE table=default.Table_adb6100fd2ea489590831f0508f53e3e [id=811a01d9de1a4185a876251bd8d10512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.829298 892 tablet_service.cc:1467] Processing CreateTablet for tablet 13683fbe9d654575b28108de8b77f943 (DEFAULT_TABLE table=default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 [id=2fe05e5dfd244de7838c0642fa521be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.830116 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a (DEFAULT_TABLE table=default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.816140 890 tablet_service.cc:1467] Processing CreateTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 (DEFAULT_TABLE table=default.Table_93c93b5d73f24f75afbcb7660706db8a [id=5abe0f9c7bb9402c941464132d4ca4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.830547 887 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.831137 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet ad9f920cde954f31acebc6d0b6965e58 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
W20250110 02:51:15.831486 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28340f0f9a2043a199c74f13755eb702 already in progress: creating tablet
I20250110 02:51:15.831600 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 1d27f9cb38434646822b6ba1fac53282 (DEFAULT_TABLE table=default.Table_01c271ca90d2495b95e92eb25502cff2 [id=5580408664d94826a2f97912fb853728]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:15.831696 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9289de0630804f9aa60fdaf5f3f15f68 already in progress: creating tablet
I20250110 02:51:15.831653 749 tablet_service.cc:1467] Processing CreateTablet for tablet f63d69d04b494a848653636c9e259105 (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.831987 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d27f9cb38434646822b6ba1fac53282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.831964 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 4e2c01af512f460daa98b19444f72172 (table default.Table_7c26cb6440db4eb8a73ac05566a26a75 [id=90791601969c442fa5bc042f2b9829af]) successfully deleted
I20250110 02:51:15.832037 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63d69d04b494a848653636c9e259105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.834604 1181 tablet_bootstrap.cc:654] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.834645 753 tablet_service.cc:1467] Processing CreateTablet for tablet 7c55788eeddb47aa8a5893703c4d5e67 (DEFAULT_TABLE table=default.Table_27ec9682796f420ebd7920d531fce2fa [id=0b98c3d632294ec4bfa3fe55e718afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.834976 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c55788eeddb47aa8a5893703c4d5e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.836481 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 (DEFAULT_TABLE table=default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.836882 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c0aaf2050f481daa8cdac4761fe705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.838632 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1d27f9cb38434646822b6ba1fac53282 (DEFAULT_TABLE table=default.Table_01c271ca90d2495b95e92eb25502cff2 [id=5580408664d94826a2f97912fb853728]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.838989 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d27f9cb38434646822b6ba1fac53282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.841539 1588 raft_consensus.cc:491] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.841681 1588 raft_consensus.cc:513] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.842353 1588 leader_election.cc:290] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:15.842737 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 (DEFAULT_TABLE table=default.Table_e04ee12d5f244793a7a44387a6243c1d [id=4ec77430bd4e44aca3a6034e4c8bc6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.842861 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2283e7ed7456dae466a8582587eb7 (DEFAULT_TABLE table=default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.843093 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae4877bba834dbb96ec9675e2aaedd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.843276 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 54b538100977456794c9519bbdffa7e7 (DEFAULT_TABLE table=default.Table_faf96e0347104a8f828bf7ba504b35ef [id=fa0cfecca12c4ddcade02cc7a553edf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.843578 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b538100977456794c9519bbdffa7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.846638 1041 tablet_service.cc:1467] Processing CreateTablet for tablet ac5dab241cd74f329a57e6f19e95a7b6 (DEFAULT_TABLE table=default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.846783 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2283e7ed7456dae466a8582587eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.847050 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5dab241cd74f329a57e6f19e95a7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.848369 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 (DEFAULT_TABLE table=default.Table_e04ee12d5f244793a7a44387a6243c1d [id=4ec77430bd4e44aca3a6034e4c8bc6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.848749 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae4877bba834dbb96ec9675e2aaedd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.848996 754 tablet_service.cc:1467] Processing CreateTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 (DEFAULT_TABLE table=default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.849339 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c0aaf2050f481daa8cdac4761fe705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.850685 755 tablet_service.cc:1467] Processing CreateTablet for tablet 54b538100977456794c9519bbdffa7e7 (DEFAULT_TABLE table=default.Table_faf96e0347104a8f828bf7ba504b35ef [id=fa0cfecca12c4ddcade02cc7a553edf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.850780 1032 tablet_service.cc:1467] Processing CreateTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.851016 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b538100977456794c9519bbdffa7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.851213 1033 tablet_service.cc:1467] Processing CreateTablet for tablet b74beb77505f472fb6b9e4a30bfe6582 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.851600 1024 tablet_service.cc:1467] Processing CreateTablet for tablet eed058931194480ca1a27130bfa8ee32 (DEFAULT_TABLE table=default.Table_6f1e9a903a2e4ecaba8ded1a3f775428 [id=78b013e77cf340089467193ef7c17819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.846647 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 213939cf13354ce380fb7cda5b38676f (DEFAULT_TABLE table=default.Table_0e2df09ed397432f8697c6baf59aaa53 [id=620801551d814277ae80ea40ecb3532b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.852710 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:15.853156 745 tablet_service.cc:1467] Processing CreateTablet for tablet ca45a5c0b3cb4b53862e51b11c4d3a00 (DEFAULT_TABLE table=default.Table_60665b68321047d3a87e132a42f6c058 [id=8c72b8cb59bc4abdb98913a891ef023a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:15.853236 711 leader_election.cc:343] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.853453 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca45a5c0b3cb4b53862e51b11c4d3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.854636 746 tablet_service.cc:1467] Processing CreateTablet for tablet 213939cf13354ce380fb7cda5b38676f (DEFAULT_TABLE table=default.Table_0e2df09ed397432f8697c6baf59aaa53 [id=620801551d814277ae80ea40ecb3532b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.854982 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213939cf13354ce380fb7cda5b38676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.855321 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7c55788eeddb47aa8a5893703c4d5e67 (DEFAULT_TABLE table=default.Table_27ec9682796f420ebd7920d531fce2fa [id=0b98c3d632294ec4bfa3fe55e718afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.856040 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c55788eeddb47aa8a5893703c4d5e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.857533 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d4f03a8c66446ea7c6d178cc7ef80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.857852 744 tablet_service.cc:1467] Processing CreateTablet for tablet ac5dab241cd74f329a57e6f19e95a7b6 (DEFAULT_TABLE table=default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.858052 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed058931194480ca1a27130bfa8ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.858188 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5dab241cd74f329a57e6f19e95a7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.858454 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74beb77505f472fb6b9e4a30bfe6582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.858774 747 tablet_service.cc:1467] Processing CreateTablet for tablet eed058931194480ca1a27130bfa8ee32 (DEFAULT_TABLE table=default.Table_6f1e9a903a2e4ecaba8ded1a3f775428 [id=78b013e77cf340089467193ef7c17819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.858852 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213939cf13354ce380fb7cda5b38676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.859087 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed058931194480ca1a27130bfa8ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.859730 743 tablet_service.cc:1467] Processing CreateTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.860137 741 tablet_service.cc:1467] Processing CreateTablet for tablet b74beb77505f472fb6b9e4a30bfe6582 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.860504 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74beb77505f472fb6b9e4a30bfe6582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.860910 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d4f03a8c66446ea7c6d178cc7ef80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.862418 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2283e7ed7456dae466a8582587eb7 (DEFAULT_TABLE table=default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.863013 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2283e7ed7456dae466a8582587eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.864780 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8835f1dac47741f69e9a48aed136cde0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.865337 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8835f1dac47741f69e9a48aed136cde0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.866530 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 461 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd" is_pre_election: true
I20250110 02:51:15.866693 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5131fbd504e04f50b92a6a1fb6d0c055"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.867193 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5131fbd504e04f50b92a6a1fb6d0c055 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.873119 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b5e3c14244418912f344b7519995d (DEFAULT_TABLE table=default.Table_48bb129a959843fcbad0b8d6344c6660 [id=c9d3b6064ff04c18a8d5e8f09a55a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.873493 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b5e3c14244418912f344b7519995d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.877796 1039 tablet_service.cc:1467] Processing CreateTablet for tablet f63d69d04b494a848653636c9e259105 (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.878881 1184 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.879076 1184 ts_tablet_manager.cc:1397] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.004s
I20250110 02:51:15.879928 1184 raft_consensus.cc:357] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.880154 1184 raft_consensus.cc:383] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.880224 1184 raft_consensus.cc:738] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.880492 1184 consensus_queue.cc:260] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.881146 1188 tablet_bootstrap.cc:492] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:15.881301 1188 ts_tablet_manager.cc:1397] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.281s user 0.010s sys 0.011s
I20250110 02:51:15.881315 1184 ts_tablet_manager.cc:1428] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:15.886447 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63d69d04b494a848653636c9e259105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.886785 1037 tablet_service.cc:1467] Processing CreateTablet for tablet ca45a5c0b3cb4b53862e51b11c4d3a00 (DEFAULT_TABLE table=default.Table_60665b68321047d3a87e132a42f6c058 [id=8c72b8cb59bc4abdb98913a891ef023a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.890309 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9395aa9b39d04ac9b6cecf106bc91499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.891184 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca45a5c0b3cb4b53862e51b11c4d3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.891604 1180 ts_tablet_manager.cc:1905] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.895964 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:15.896154 1434 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10; no voters: 8bc934cc708045e9858ee8f8e64660cd, 8fca9a92f47145fcaaf4f65038d6f558
I20250110 02:51:15.896551 1620 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 02:51:15.897439 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c5d78661dad44d9f85ba6c91486185de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.898128 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5d78661dad44d9f85ba6c91486185de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.901867 1188 raft_consensus.cc:357] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.902122 1188 raft_consensus.cc:383] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.902642 1188 raft_consensus.cc:738] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.902896 1188 consensus_queue.cc:260] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:15.906683 517 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eed058931194480ca1a27130bfa8ee32 consensus state has no leader
I20250110 02:51:15.908183 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:15.911804 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1608d90ceb64602889b5705ba4ab55d consensus state has no leader
I20250110 02:51:15.912031 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b5e3c14244418912f344b7519995d (DEFAULT_TABLE table=default.Table_48bb129a959843fcbad0b8d6344c6660 [id=c9d3b6064ff04c18a8d5e8f09a55a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.913373 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eadb829d657346eabcb7d2aa34b33fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.913601 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13683fbe9d654575b28108de8b77f943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.913739 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.914130 520 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]). Expected version 1 got 0
I20250110 02:51:15.914909 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.914564 520 catalog_manager.cc:5526] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:15.915946 520 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:15.916273 1188 ts_tablet_manager.cc:1428] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250110 02:51:15.916548 1188 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:15.916908 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b5e3c14244418912f344b7519995d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.918164 1184 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.919209 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.919384 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.919534 887 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.921185 1184 tablet_bootstrap.cc:654] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.924139 1180 ts_tablet_manager.cc:1918] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:15.924259 1180 log.cc:1198] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/514704cd935b44e8b760221224332509
I20250110 02:51:15.924700 1180 ts_tablet_manager.cc:1939] T 514704cd935b44e8b760221224332509 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.925997 1180 tablet_replica.cc:331] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.926208 1180 raft_consensus.cc:2238] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.926928 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.927563 1184 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:15.927676 1184 ts_tablet_manager.cc:1397] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20250110 02:51:15.928380 1184 raft_consensus.cc:357] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.928552 1184 raft_consensus.cc:383] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.928630 1184 raft_consensus.cc:738] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.928831 1180 raft_consensus.cc:2267] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.928848 1184 consensus_queue.cc:260] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:15.929324 1184 ts_tablet_manager.cc:1428] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:15.929334 1180 ts_tablet_manager.cc:1905] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:15.932443 1184 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:15.932628 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 514704cd935b44e8b760221224332509 (table default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]) successfully deleted
I20250110 02:51:15.934046 1184 tablet_bootstrap.cc:654] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.936388 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c8632cb324c745eca03df9733d23a9a7" } new_table_name: "default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa" modify_external_catalogs: true
I20250110 02:51:15.938234 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_424f77049b1645eb850dfb8e41db29bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.938833 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_424f77049b1645eb850dfb8e41db29bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.938776 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_68d7c1c889464468adcd9153caef7ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.939401 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68d7c1c889464468adcd9153caef7ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.940542 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_042ec589caf34c7487d9fa394cb51419" } new_table_name: "default.Table_8d7fe23e0afe4dfdac17778b492f2c55" modify_external_catalogs: true
I20250110 02:51:15.942454 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:15.944460 1180 ts_tablet_manager.cc:1918] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:15.944602 1180 log.cc:1198] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1dfc062b5fcb42c3bce6dfaed54b3869
I20250110 02:51:15.944851 894 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b5e3c14244418912f344b7519995d (DEFAULT_TABLE table=default.Table_48bb129a959843fcbad0b8d6344c6660 [id=c9d3b6064ff04c18a8d5e8f09a55a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.944931 1180 ts_tablet_manager.cc:1939] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:15.946305 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b5e3c14244418912f344b7519995d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.946306 1181 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:15.946449 1181 ts_tablet_manager.cc:1397] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.255s user 0.001s sys 0.008s
I20250110 02:51:15.947254 1181 raft_consensus.cc:357] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.947474 1181 raft_consensus.cc:383] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:15.947559 1181 raft_consensus.cc:738] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:15.947996 891 tablet_service.cc:1467] Processing CreateTablet for tablet f63d69d04b494a848653636c9e259105 (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.947813 1181 consensus_queue.cc:260] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.949982 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63d69d04b494a848653636c9e259105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.951058 882 tablet_service.cc:1467] Processing CreateTablet for tablet 7c55788eeddb47aa8a5893703c4d5e67 (DEFAULT_TABLE table=default.Table_27ec9682796f420ebd7920d531fce2fa [id=0b98c3d632294ec4bfa3fe55e718afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.951210 1181 ts_tablet_manager.cc:1428] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:15.951761 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1d27f9cb38434646822b6ba1fac53282 (DEFAULT_TABLE table=default.Table_01c271ca90d2495b95e92eb25502cff2 [id=5580408664d94826a2f97912fb853728]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.952291 1188 tablet_bootstrap.cc:654] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.952515 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d27f9cb38434646822b6ba1fac53282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.952785 1181 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:15.954761 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
W20250110 02:51:15.955101 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 95b14d7592024a34baf7756d92e546d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95b14d7592024a34baf7756d92e546d9 already in progress: creating tablet
I20250110 02:51:15.954860 1180 tablet_replica.cc:331] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:15.955484 895 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 (DEFAULT_TABLE table=default.Table_e04ee12d5f244793a7a44387a6243c1d [id=4ec77430bd4e44aca3a6034e4c8bc6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.955768 1180 raft_consensus.cc:2238] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.956282 1180 raft_consensus.cc:2267] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:15.955899 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
W20250110 02:51:15.956750 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d970e2fe72b4727968ca605b3e772cc already in progress: creating tablet
W20250110 02:51:15.959204 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
W20250110 02:51:15.959404 709 leader_election.cc:343] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:15.959574 709 leader_election.cc:304] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:15.959482 1180 ts_tablet_manager.cc:1905] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:15.959797 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet da8f247c862e4cb68452e230857ee6b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da8f247c862e4cb68452e230857ee6b0 already in progress: creating tablet
I20250110 02:51:15.959754 1181 tablet_bootstrap.cc:654] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.960055 1588 raft_consensus.cc:2743] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:15.961323 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:15.961793 709 leader_election.cc:343] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:15.960780 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because the tablet was not found. No further retry: Not found: Tablet not found: 5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:15.961974 709 leader_election.cc:304] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:15.962181 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because the tablet was not found. No further retry: Not found: Tablet not found: 213939cf13354ce380fb7cda5b38676f
I20250110 02:51:15.962327 1588 raft_consensus.cc:2743] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:15.962514 510 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because the tablet was not found. No further retry: Not found: Tablet not found: a7d4f03a8c66446ea7c6d178cc7ef80a
I20250110 02:51:15.962709 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c55788eeddb47aa8a5893703c4d5e67. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:15.962750 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e05a7554a14805bdcbd1824a5bd9d5 already in progress: creating tablet
W20250110 02:51:15.963097 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2163c343a694f5ca04ac9f7610d25e3 already in progress: creating tablet
I20250110 02:51:15.963194 879 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2283e7ed7456dae466a8582587eb7 (DEFAULT_TABLE table=default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:15.963285 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
W20250110 02:51:15.963480 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 already in progress: creating tablet
I20250110 02:51:15.963748 887 tablet_service.cc:1467] Processing CreateTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 (DEFAULT_TABLE table=default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.963922 896 tablet_service.cc:1467] Processing CreateTablet for tablet ca45a5c0b3cb4b53862e51b11c4d3a00 (DEFAULT_TABLE table=default.Table_60665b68321047d3a87e132a42f6c058 [id=8c72b8cb59bc4abdb98913a891ef023a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.964939 880 tablet_service.cc:1467] Processing CreateTablet for tablet 213939cf13354ce380fb7cda5b38676f (DEFAULT_TABLE table=default.Table_0e2df09ed397432f8697c6baf59aaa53 [id=620801551d814277ae80ea40ecb3532b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.967000 883 tablet_service.cc:1467] Processing CreateTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a (DEFAULT_TABLE table=default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.968372 893 tablet_service.cc:1467] Processing CreateTablet for tablet b74beb77505f472fb6b9e4a30bfe6582 (DEFAULT_TABLE table=default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.963685 898 tablet_service.cc:1467] Processing CreateTablet for tablet 54b538100977456794c9519bbdffa7e7 (DEFAULT_TABLE table=default.Table_faf96e0347104a8f828bf7ba504b35ef [id=fa0cfecca12c4ddcade02cc7a553edf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.971484 886 tablet_service.cc:1467] Processing CreateTablet for tablet ac5dab241cd74f329a57e6f19e95a7b6 (DEFAULT_TABLE table=default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.972559 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae4877bba834dbb96ec9675e2aaedd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.972793 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2283e7ed7456dae466a8582587eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.972940 884 tablet_service.cc:1467] Processing CreateTablet for tablet eed058931194480ca1a27130bfa8ee32 (DEFAULT_TABLE table=default.Table_6f1e9a903a2e4ecaba8ded1a3f775428 [id=78b013e77cf340089467193ef7c17819]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:15.973222 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed058931194480ca1a27130bfa8ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.974356 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca45a5c0b3cb4b53862e51b11c4d3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.984851 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5dab241cd74f329a57e6f19e95a7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.985827 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7d4f03a8c66446ea7c6d178cc7ef80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.986065 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:15.987520 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:15.987924 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213939cf13354ce380fb7cda5b38676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.988143 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74beb77505f472fb6b9e4a30bfe6582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:15.989713 1635 raft_consensus.cc:491] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:15.989887 1635 raft_consensus.cc:513] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:15.990641 1635 leader_election.cc:290] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:15.991566 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:15.991604 996 leader_election.cc:343] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 48cb20d47ace4fd49f9dc4e526cab702
I20250110 02:51:15.991819 1179 tablet_replica.cc:331] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:15.991959 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:15.992033 1179 raft_consensus.cc:2238] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:15.992205 1179 raft_consensus.cc:2267] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:15.993376 1179 ts_tablet_manager.cc:1905] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.000507 1179 ts_tablet_manager.cc:1918] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.000622 1179 log.cc:1198] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:16.000959 1179 ts_tablet_manager.cc:1939] T 5af3d61366b2432db53b18f2b7bb49d9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.000900 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_db04ddcfa7864a5a82f05df887a858aa" } new_table_name: "default.Table_350d7434def5474991dcc142163cf175" modify_external_catalogs: true
I20250110 02:51:16.001336 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_68a6c97dd17c47b28f987d6e8e778e37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.001804 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a6c97dd17c47b28f987d6e8e778e37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.002319 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f10221708fa5417d8300ada3f8eb0817" } modify_external_catalogs: true
I20250110 02:51:16.002511 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_f10221708fa5417d8300ada3f8eb0817" } modify_external_catalogs: true
I20250110 02:51:16.003294 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cce012e47fbc46f4807cb0cd9adc5568" } modify_external_catalogs: true
I20250110 02:51:16.003448 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_cce012e47fbc46f4807cb0cd9adc5568" } modify_external_catalogs: true
I20250110 02:51:16.003700 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7431ea53e3704aebb4d79e0a0c4045f5" } modify_external_catalogs: true
I20250110 02:51:16.003840 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_7431ea53e3704aebb4d79e0a0c4045f5" } modify_external_catalogs: true
I20250110 02:51:16.004542 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_82157dfbcfe64e759c6847722284ee3f" } modify_external_catalogs: true
I20250110 02:51:16.004709 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_82157dfbcfe64e759c6847722284ee3f" } modify_external_catalogs: true
I20250110 02:51:16.007225 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb20d47ace4fd49f9dc4e526cab702" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.008165 994 leader_election.cc:343] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.008291 994 leader_election.cc:304] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:16.008684 1635 raft_consensus.cc:2743] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:16.011126 1184 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.011415 1184 ts_tablet_manager.cc:1397] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.079s user 0.018s sys 0.031s
I20250110 02:51:16.011893 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c0aaf2050f481daa8cdac4761fe705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.013304 1184 raft_consensus.cc:357] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.013581 1184 raft_consensus.cc:383] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.013716 1184 raft_consensus.cc:738] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.013978 756 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.014231 1179 tablet_replica.cc:331] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:16.014015 1184 consensus_queue.cc:260] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.014437 1179 raft_consensus.cc:2238] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.016573 1179 raft_consensus.cc:2267] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.017021 1184 ts_tablet_manager.cc:1428] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250110 02:51:16.017380 1184 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.018141 1179 ts_tablet_manager.cc:1905] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.019148 1184 tablet_bootstrap.cc:654] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.022112 1180 ts_tablet_manager.cc:1918] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.024648 1180 log.cc:1198] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c65704c9d486444ebe8e4ffd4580f395
I20250110 02:51:16.025321 1180 ts_tablet_manager.cc:1939] T c65704c9d486444ebe8e4ffd4580f395 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.027273 1180 tablet_replica.cc:331] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.028069 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b538100977456794c9519bbdffa7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.029234 1180 raft_consensus.cc:2238] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.029402 1180 raft_consensus.cc:2267] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.029937 1180 ts_tablet_manager.cc:1905] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.032719 1179 ts_tablet_manager.cc:1918] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.033320 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 5af3d61366b2432db53b18f2b7bb49d9 (table default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]) successfully deleted
I20250110 02:51:16.033377 1179 log.cc:1198] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:16.033674 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c65704c9d486444ebe8e4ffd4580f395 (table default.Table_420beb896535455a87ee0a8c6a6049d4 [id=1d00dbb015e142329070adad93cf1bcb]) successfully deleted
I20250110 02:51:16.034994 1179 ts_tablet_manager.cc:1939] T dad0d561a8ef498ba13bc813d4cd82fa P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.037598 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet dad0d561a8ef498ba13bc813d4cd82fa (table default.Table_5bfa749055ab45479cf96da68e537ece [id=488d922a6b604812af65ff6b4d9c6606]) successfully deleted
I20250110 02:51:16.035816 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 514704cd935b44e8b760221224332509 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.050789 1180 ts_tablet_manager.cc:1918] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.051405 1180 log.cc:1198] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/48cb20d47ace4fd49f9dc4e526cab702
I20250110 02:51:16.052129 1180 ts_tablet_manager.cc:1939] T 48cb20d47ace4fd49f9dc4e526cab702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.053886 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 48cb20d47ace4fd49f9dc4e526cab702 (table default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]) successfully deleted
I20250110 02:51:16.054548 1178 tablet_replica.cc:331] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:16.055770 1180 tablet_replica.cc:331] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.056403 1180 raft_consensus.cc:2238] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:16.057183 1180 raft_consensus.cc:2267] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.059311 1180 ts_tablet_manager.cc:1905] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.061758 1178 raft_consensus.cc:2238] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.063763 1178 raft_consensus.cc:2267] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.064509 1178 ts_tablet_manager.cc:1905] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.064880 916 raft_consensus.cc:3054] T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.076478 1181 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.076650 1181 ts_tablet_manager.cc:1397] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.012s
I20250110 02:51:16.077452 1181 raft_consensus.cc:357] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.077682 1181 raft_consensus.cc:383] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.077764 1181 raft_consensus.cc:738] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.078008 1181 consensus_queue.cc:260] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.081241 1180 ts_tablet_manager.cc:1918] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:16.082901 1180 log.cc:1198] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c0edfa33e30042edad853217aa761bd5
I20250110 02:51:16.083467 1180 ts_tablet_manager.cc:1939] T c0edfa33e30042edad853217aa761bd5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.085203 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c0edfa33e30042edad853217aa761bd5 (table default.Table_306f9f127f75420882d16d81923bf9f5 [id=4f08fffc4d93445687c1abf05f669666]) successfully deleted
W20250110 02:51:16.086660 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76833ce515cf4ee9bc8fff0f626af220 already in progress: creating tablet
I20250110 02:51:16.086884 1181 ts_tablet_manager.cc:1428] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250110 02:51:16.087500 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c09235548b5d4e10a01cbe05d1c94841 consensus state has no leader
W20250110 02:51:16.087549 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 already in progress: creating tablet
W20250110 02:51:16.088327 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
I20250110 02:51:16.088510 1184 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.088675 1184 ts_tablet_manager.cc:1397] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.031s
W20250110 02:51:16.089172 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65285bce60fd4015b085deb7442eea6e already in progress: creating tablet
I20250110 02:51:16.089362 1181 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.089525 1184 raft_consensus.cc:357] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.089756 1184 raft_consensus.cc:383] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.089852 1184 raft_consensus.cc:738] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.090121 1184 consensus_queue.cc:260] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:16.090634 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0d4d6e02b1d4334a9993d320217f1f8 consensus state has no leader
W20250110 02:51:16.091354 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
W20250110 02:51:16.092044 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2163c343a694f5ca04ac9f7610d25e3 already in progress: creating tablet
I20250110 02:51:16.092365 1184 ts_tablet_manager.cc:1428] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250110 02:51:16.092315 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9ca42c50470d4cebaf832606cb958d23" } modify_external_catalogs: true
I20250110 02:51:16.092497 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9ca42c50470d4cebaf832606cb958d23" } modify_external_catalogs: true
I20250110 02:51:16.092547 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.092662 1184 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:16.094784 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
I20250110 02:51:16.095219 1180 tablet_replica.cc:331] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.095417 1180 raft_consensus.cc:2238] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.095561 1180 raft_consensus.cc:2267] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.095834 1184 tablet_bootstrap.cc:654] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.096037 1180 ts_tablet_manager.cc:1905] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.101660 754 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.101953 1179 tablet_replica.cc:331] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:16.102147 1179 raft_consensus.cc:2238] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.102345 1179 raft_consensus.cc:2267] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.103379 1179 ts_tablet_manager.cc:1905] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.107903 1181 tablet_bootstrap.cc:654] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.110168 1179 ts_tablet_manager.cc:1918] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.110353 1179 log.cc:1198] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:16.110786 1179 ts_tablet_manager.cc:1939] T da8f247c862e4cb68452e230857ee6b0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.112058 524 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_faf96e0347104a8f828bf7ba504b35ef" } new_table_name: "default.Table_c8df69267ed3452192228c2106e18a47" modify_external_catalogs: true
I20250110 02:51:16.111608 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.114202 526 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_105ea910a3c447a6abb0c0919ab8ee06" } new_table_name: "default.Table_c2f177ce9d6a420bbc77b33929ccaa7a" modify_external_catalogs: true
I20250110 02:51:16.114856 1179 tablet_replica.cc:331] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:16.114871 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet da8f247c862e4cb68452e230857ee6b0 (table default.Table_90e43feb19a343f6b3c9a20876ec8c23 [id=e2a6764b09814b9cb1367abb1a4f5ca9]) successfully deleted
I20250110 02:51:16.115096 1179 raft_consensus.cc:2238] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.115236 1179 raft_consensus.cc:2267] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.115952 1179 ts_tablet_manager.cc:1905] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.120026 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfc062b5fcb42c3bce6dfaed54b3869 with delete_type TABLET_DATA_DELETED (replaced by tablet 585320c0970c4097b195caaae33789b5) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.132117 1180 ts_tablet_manager.cc:1918] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.133313 1179 ts_tablet_manager.cc:1918] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.133543 1179 log.cc:1198] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:16.133983 1179 ts_tablet_manager.cc:1939] T 2ef72e05f8ed4d46b759c7a04497653b P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.135114 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2ef72e05f8ed4d46b759c7a04497653b (table default.Table_e298b60e8778429bb52cf546d58fa2d9 [id=62df87bfc41d477fb6044eff0ed5c3ea]) successfully deleted
I20250110 02:51:16.135416 1180 log.cc:1198] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:16.135891 1180 ts_tablet_manager.cc:1939] T 2abd85a070f74e9bb5dc4026935edcb5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.136881 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2abd85a070f74e9bb5dc4026935edcb5 (table default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]) successfully deleted
W20250110 02:51:16.137549 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
W20250110 02:51:16.138121 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
W20250110 02:51:16.138712 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
I20250110 02:51:16.139088 1181 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.139220 1181 ts_tablet_manager.cc:1397] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
W20250110 02:51:16.139295 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d4f03a8c66446ea7c6d178cc7ef80a already in progress: creating tablet
W20250110 02:51:16.139845 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e05a7554a14805bdcbd1824a5bd9d5 already in progress: creating tablet
I20250110 02:51:16.139989 1181 raft_consensus.cc:357] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.140175 1181 raft_consensus.cc:383] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.140259 1181 raft_consensus.cc:738] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:16.140525 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 already in progress: creating tablet
I20250110 02:51:16.140508 1181 consensus_queue.cc:260] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.141021 1181 ts_tablet_manager.cc:1428] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:16.143914 1181 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.146796 1180 tablet_replica.cc:331] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.147038 1180 raft_consensus.cc:2238] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.147259 1180 raft_consensus.cc:2267] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.148013 1180 ts_tablet_manager.cc:1905] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.153975 1181 tablet_bootstrap.cc:654] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.156103 1180 ts_tablet_manager.cc:1918] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.157025 1180 log.cc:1198] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:16.157545 1180 ts_tablet_manager.cc:1939] T 8940c201aa104b0b8bc8d866859fe8ab P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.158896 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8940c201aa104b0b8bc8d866859fe8ab (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:16.161458 1180 tablet_replica.cc:331] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.161867 1180 raft_consensus.cc:2238] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.162271 1180 raft_consensus.cc:2267] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.163086 1180 ts_tablet_manager.cc:1905] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.174140 1180 ts_tablet_manager.cc:1918] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.174655 1180 log.cc:1198] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2e10e7654cec495bb532fb00e73c020d
I20250110 02:51:16.175293 1181 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.175499 1181 ts_tablet_manager.cc:1397] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.000s
I20250110 02:51:16.176403 1181 raft_consensus.cc:357] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.176680 1181 raft_consensus.cc:383] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.176795 1181 raft_consensus.cc:738] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.177196 1181 consensus_queue.cc:260] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.179649 1181 ts_tablet_manager.cc:1428] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:16.180094 1635 consensus_queue.cc:1035] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:16.179986 1181 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.183264 1181 tablet_bootstrap.cc:654] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.184887 1180 ts_tablet_manager.cc:1939] T 2e10e7654cec495bb532fb00e73c020d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.187100 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2e10e7654cec495bb532fb00e73c020d (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]) successfully deleted
I20250110 02:51:16.190834 1181 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.191018 1181 ts_tablet_manager.cc:1397] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20250110 02:51:16.191843 1181 raft_consensus.cc:357] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.192087 1181 raft_consensus.cc:383] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.192219 1181 raft_consensus.cc:738] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.192492 1181 consensus_queue.cc:260] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.193135 1181 ts_tablet_manager.cc:1428] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:16.193401 1181 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.195211 1545 consensus_queue.cc:1035] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:16.203775 1635 consensus_queue.cc:1035] T 1cc0532c94eb401e9d299e79556b4cf1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:16.204473 1181 tablet_bootstrap.cc:654] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:15.553936 1481 rpcz_store.cc:276] 0110 02:51:14.429384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:14.429437 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:14.638702 (+209265us) raft_consensus.cc:1466] Updating replica for 439 ops
0110 02:51:14.768831 (+130129us) raft_consensus.cc:1529] Early marking committed up to index 35
0110 02:51:14.797987 (+ 29156us) raft_consensus.cc:1534] Triggering prepare for 426 ops
0110 02:51:14.829023 (+ 31036us) write_op.cc:270] Start()
0110 02:51:14.829093 (+ 70us) write_op.cc:276] Timestamp: P: 1736477468631175 usec, L: 0
0110 02:51:14.829437 (+ 344us) write_op.cc:270] Start()
0110 02:51:14.829465 (+ 28us) write_op.cc:276] Timestamp: P: 1736477468632009 usec, L: 0
0110 02:51:14.829618 (+ 153us) write_op.cc:270] Start()
0110 02:51:14.829641 (+ 23us) write_op.cc:276] Timestamp: P: 1736477468632504 usec, L: 0
0110 02:51:14.830701 (+ 1060us) write_op.cc:270] Start()
0110 02:51:14.830745 (+ 44us) write_op.cc:276] Timestamp: P: 1736477468632955 usec, L: 0
0110 02:51:14.841257 (+ 10512us) write_op.cc:270] Start()
0110 02:51:14.841285 (+ 28us) write_op.cc:276] Timestamp: P: 1736477468633462 usec, L: 0
0110 02:51:14.841388 (+ 103us) write_op.cc:270] Start()
0110 02:51:14.841402 (+ 14us) write_op.cc:276] Timestamp: P: 1736477468633901 usec, L: 0
0110 02:51:14.841499 (+ 97us) write_op.cc:270] Start()
0110 02:51:14.841526 (+ 27us) write_op.cc:276] Timestamp: P: 1736477468634309 usec, L: 0
0110 02:51:14.841611 (+ 85us) write_op.cc:270] Start()
0110 02:51:14.841624 (+ 13us) write_op.cc:276] Timestamp: P: 1736477468642523 usec, L: 0
0110 02:51:14.841716 (+ 92us) write_op.cc:270] Start()
0110 02:51:14.841730 (+ 14us) write_op.cc:276] Timestamp: P: 1736477468666905 usec, L: 0
0110 02:51:14.841822 (+ 92us) write_op.cc:270] Start()
0110 02:51:14.841835 (+ 13us) write_op.cc:276] Timestamp: P: 1736477468680819 usec, L: 0
0110 02:51:14.841923 (+ 88us) write_op.cc:270] Start()
0110 02:51:14.841938 (+ 15us) write_op.cc:276] Timestamp: P: 1736477468701986 usec, L: 0
0110 02:51:14.842028 (+ 90us) write_op.cc:270] Start()
0110 02:51:14.842042 (+ 14us) write_op.cc:276] Timestamp: P: 1736477468703300 usec, L: 0
0110 02:51:14.842123 (+ 81us) write_op.cc:270] Start()
0110 02:51:14.842136 (+ 13us) write_op.cc:276] Timestamp: P: 1736477468703994 usec, L: 0
0110 02:51:14.842236 (+ 100us) write_op.cc:270] Start()
0110 02:51:14.842249 (+ 13us) write_op.cc:276] Timestamp: P: 1736477468704907 usec, L: 0
0110 02:51:14.842336 (+ 87us) write_op.cc:270] Start()
0110 02:51:14.842349 (+ 13us) write_op.cc:276] Timestamp: P: 1736477468725099 usec, L: 0
0110 02:51:14.851228 (+ 8879us) write_op.cc:270] Start()
0110 02:51:14.851263 (+ 35us) write_op.cc:276] Timestamp: P: 1736477468725964 usec, L: 0
0110 02:51:14.851469 (+ 206us) write_op.cc:270] Start()
0110 02:51:14.851488 (+ 19us) write_op.cc:276] Timestamp: P: 1736477468729930 usec, L: 0
0110 02:51:14.851623 (+ 135us) write_op.cc:270] Start()
0110 02:51:14.851642 (+ 19us) write_op.cc:276] Timestamp: P: 1736477468746313 usec, L: 0
0110 02:51:14.851760 (+ 118us) write_op.cc:270] Start()
0110 02:51:14.851782 (+ 22us) write_op.cc:276] Timestamp: P: 1736477468760721 usec, L: 0
0110 02:51:14.851907 (+ 125us) write_op.cc:270] Start()
0110 02:51:14.851936 (+ 29us) write_op.cc:276] Timestamp: P: 1736477468764060 usec, L: 0
0110 02:51:14.852083 (+ 147us) write_op.cc:270] Start()
0110 02:51:14.852103 (+ 20us) write_op.cc:276] Timestamp: P: 1736477468780610 usec, L: 0
0110 02:51:14.852233 (+ 130us) write_op.cc:270] Start()
0110 02:51:14.852249 (+ 16us) write_op.cc:276] Timestamp: P: 1736477468798388 usec, L: 0
0110 02:51:14.852359 (+ 110us) write_op.cc:270] Start()
0110 02:51:14.852376 (+ 17us) write_op.cc:276] Timestamp: P: 1736477468804932 usec, L: 0
0110 02:51:14.852491 (+ 115us) write_op.cc:270] Start()
0110 02:51:14.852518 (+ 27us) write_op.cc:276] Timestamp: P: 1736477468840866 usec, L: 0
0110 02:51:14.852685 (+ 167us) write_op.cc:270] Start()
0110 02:51:14.852711 (+ 26us) write_op.cc:276] Timestamp: P: 1736477468847954 usec, L: 0
0110 02:51:14.852866 (+ 155us) write_op.cc:270] Start()
0110 02:51:14.852885 (+ 19us) write_op.cc:276] Timestamp: P: 1736477468851538 usec, L: 0
0110 02:51:14.853037 (+ 152us) write_op.cc:270] Start()
0110 02:51:14.853054 (+ 17us) write_op.cc:276] Timestamp: P: 1736477468852024 usec, L: 0
0110 02:51:14.853169 (+ 115us) write_op.cc:270] Start()
0110 02:51:14.853183 (+ 14us) write_op.cc:276] Timestamp: P: 1736477468890107 usec, L: 0
0110 02:51:14.853288 (+ 105us) write_op.cc:270] Start()
0110 02:51:14.853303 (+ 15us) write_op.cc:276] Timestamp: P: 1736477468897874 usec, L: 0
0110 02:51:14.853449 (+ 146us) write_op.cc:270] Start()
0110 02:51:14.853465 (+ 16us) write_op.cc:276] Timestamp: P: 1736477468899019 usec, L: 0
0110 02:51:14.853584 (+ 119us) write_op.cc:270] Start()
0110 02:51:14.853607 (+ 23us) write_op.cc:276] Timestamp: P: 1736477468899590 usec, L: 0
0110 02:51:14.853745 (+ 138us) write_op.cc:270] Start()
0110 02:51:14.853767 (+ 22us) write_op.cc:276] Timestamp: P: 1736477468921476 usec, L: 0
0110 02:51:14.853911 (+ 144us) write_op.cc:270] Start()
0110 02:51:14.853941 (+ 30us) write_op.cc:276] Timestamp: P: 1736477468942306 usec, L: 0
0110 02:51:14.870057 (+ 16116us) write_op.cc:270] Start()
0110 02:51:14.870093 (+ 36us) write_op.cc:276] Timestamp: P: 1736477468943084 usec, L: 0
0110 02:51:14.870244 (+ 151us) write_op.cc:270] Start()
0110 02:51:14.870260 (+ 16us) write_op.cc:276] Timestamp: P: 1736477468943734 usec, L: 0
0110 02:51:14.870364 (+ 104us) write_op.cc:270] Start()
0110 02:51:14.870398 (+ 34us) write_op.cc:276] Timestamp: P: 1736477468978757 usec, L: 0
0110 02:51:14.870515 (+ 117us) write_op.cc:270] Start()
0110 02:51:14.870533 (+ 18us) write_op.cc:276] Timestamp: P: 1736477468979394 usec, L: 0
0110 02:51:14.870727 (+ 194us) write_op.cc:270] Start()
0110 02:51:14.870750 (+ 23us) write_op.cc:276] Timestamp: P: 1736477468991456 usec, L: 0
0110 02:51:14.870898 (+ 148us) write_op.cc:270] Start()
0110 02:51:14.870927 (+ 29us) write_op.cc:276] Timestamp: P: 1736477468992284 usec, L: 0
0110 02:51:14.871071 (+ 144us) write_op.cc:270] Start()
0110 02:51:14.871093 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469006653 usec, L: 0
0110 02:51:14.871240 (+ 147us) write_op.cc:270] Start()
0110 02:51:14.871259 (+ 19us) write_op.cc:276] Timestamp: P: 1736477469018139 usec, L: 0
0110 02:51:14.871384 (+ 125us) write_op.cc:270] Start()
0110 02:51:14.871405 (+ 21us) write_op.cc:276] Timestamp: P: 1736477469019179 usec, L: 0
0110 02:51:14.871544 (+ 139us) write_op.cc:270] Start()
0110 02:51:14.871567 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469035548 usec, L: 0
0110 02:51:14.871713 (+ 146us) write_op.cc:270] Start()
0110 02:51:14.871736 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469036233 usec, L: 0
0110 02:51:14.871911 (+ 175us) write_op.cc:270] Start()
0110 02:51:14.871941 (+ 30us) write_op.cc:276] Timestamp: P: 1736477469081120 usec, L: 0
0110 02:51:14.872068 (+ 127us) write_op.cc:270] Start()
0110 02:51:14.872088 (+ 20us) write_op.cc:276] Timestamp: P: 1736477469095996 usec, L: 0
0110 02:51:14.872207 (+ 119us) write_op.cc:270] Start()
0110 02:51:14.872224 (+ 17us) write_op.cc:276] Timestamp: P: 1736477469099472 usec, L: 0
0110 02:51:14.877218 (+ 4994us) write_op.cc:270] Start()
0110 02:51:14.877241 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469100053 usec, L: 0
0110 02:51:14.877393 (+ 152us) write_op.cc:270] Start()
0110 02:51:14.877415 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469134896 usec, L: 0
0110 02:51:14.877582 (+ 167us) write_op.cc:270] Start()
0110 02:51:14.877605 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469147888 usec, L: 0
0110 02:51:14.877727 (+ 122us) write_op.cc:270] Start()
0110 02:51:14.877752 (+ 25us) write_op.cc:276] Timestamp: P: 1736477469149876 usec, L: 0
0110 02:51:14.879471 (+ 1719us) write_op.cc:270] Start()
0110 02:51:14.879494 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469154225 usec, L: 0
0110 02:51:14.879621 (+ 127us) write_op.cc:270] Start()
0110 02:51:14.879637 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469170748 usec, L: 0
0110 02:51:14.879767 (+ 130us) write_op.cc:270] Start()
0110 02:51:14.879784 (+ 17us) write_op.cc:276] Timestamp: P: 1736477469178991 usec, L: 0
0110 02:51:14.879886 (+ 102us) write_op.cc:270] Start()
0110 02:51:14.879902 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469183414 usec, L: 0
0110 02:51:14.880021 (+ 119us) write_op.cc:270] Start()
0110 02:51:14.880037 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469184182 usec, L: 0
0110 02:51:14.880163 (+ 126us) write_op.cc:270] Start()
0110 02:51:14.880181 (+ 18us) write_op.cc:276] Timestamp: P: 1736477469191090 usec, L: 0
0110 02:51:14.880281 (+ 100us) write_op.cc:270] Start()
0110 02:51:14.880295 (+ 14us) write_op.cc:276] Timestamp: P: 1736477469198946 usec, L: 0
0110 02:51:14.880415 (+ 120us) write_op.cc:270] Start()
0110 02:51:14.880431 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469199499 usec, L: 0
0110 02:51:14.880537 (+ 106us) write_op.cc:270] Start()
0110 02:51:14.880571 (+ 34us) write_op.cc:276] Timestamp: P: 1736477469200244 usec, L: 0
0110 02:51:14.880672 (+ 101us) write_op.cc:270] Start()
0110 02:51:14.880687 (+ 15us) write_op.cc:276] Timestamp: P: 1736477469238996 usec, L: 0
0110 02:51:14.880816 (+ 129us) write_op.cc:270] Start()
0110 02:51:14.880839 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469314370 usec, L: 0
0110 02:51:14.880998 (+ 159us) write_op.cc:270] Start()
0110 02:51:14.881020 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469315008 usec, L: 0
0110 02:51:14.881188 (+ 168us) write_op.cc:270] Start()
0110 02:51:14.881209 (+ 21us) write_op.cc:276] Timestamp: P: 1736477469319011 usec, L: 0
0110 02:51:14.881395 (+ 186us) write_op.cc:270] Start()
0110 02:51:14.881420 (+ 25us) write_op.cc:276] Timestamp: P: 1736477469319476 usec, L: 0
0110 02:51:14.881568 (+ 148us) write_op.cc:270] Start()
0110 02:51:14.881613 (+ 45us) write_op.cc:276] Timestamp: P: 1736477469319957 usec, L: 0
0110 02:51:14.881767 (+ 154us) write_op.cc:270] Start()
0110 02:51:14.881786 (+ 19us) write_op.cc:276] Timestamp: P: 1736477469320591 usec, L: 0
0110 02:51:14.881951 (+ 165us) write_op.cc:270] Start()
0110 02:51:14.889817 (+ 7866us) write_op.cc:276] Timestamp: P: 1736477469334057 usec, L: 0
0110 02:51:14.890021 (+ 204us) write_op.cc:270] Start()
0110 02:51:14.890046 (+ 25us) write_op.cc:276] Timestamp: P: 1736477469334587 usec, L: 0
0110 02:51:14.890191 (+ 145us) write_op.cc:270] Start()
0110 02:51:14.890211 (+ 20us) write_op.cc:276] Timestamp: P: 1736477469345110 usec, L: 0
0110 02:51:14.890361 (+ 150us) write_op.cc:270] Start()
0110 02:51:14.890406 (+ 45us) write_op.cc:276] Timestamp: P: 1736477469365794 usec, L: 0
0110 02:51:14.890554 (+ 148us) write_op.cc:270] Start()
0110 02:51:14.890579 (+ 25us) write_op.cc:276] Timestamp: P: 1736477469366833 usec, L: 0
0110 02:51:14.890769 (+ 190us) write_op.cc:270] Start()
0110 02:51:14.890786 (+ 17us) write_op.cc:276] Timestamp: P: 1736477469393076 usec, L: 0
0110 02:51:14.890903 (+ 117us) write_op.cc:270] Start()
0110 02:51:14.890932 (+ 29us) write_op.cc:276] Timestamp: P: 1736477469393658 usec, L: 0
0110 02:51:14.901169 (+ 10237us) write_op.cc:270] Start()
0110 02:51:14.901206 (+ 37us) write_op.cc:276] Timestamp: P: 1736477469410059 usec, L: 0
0110 02:51:14.901376 (+ 170us) write_op.cc:270] Start()
0110 02:51:14.901395 (+ 19us) write_op.cc:276] Timestamp: P: 1736477469414118 usec, L: 0
0110 02:51:14.901512 (+ 117us) write_op.cc:270] Start()
0110 02:51:14.901529 (+ 17us) write_op.cc:276] Timestamp: P: 1736477469475029 usec, L: 0
0110 02:51:14.901641 (+ 112us) write_op.cc:270] Start()
0110 02:51:14.901664 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469498675 usec, L: 0
0110 02:51:14.901780 (+ 116us) write_op.cc:270] Start()
0110 02:51:14.901796 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469567266 usec, L: 0
0110 02:51:14.901901 (+ 105us) write_op.cc:270] Start()
0110 02:51:14.901916 (+ 15us) write_op.cc:276] Timestamp: P: 1736477469568496 usec, L: 0
0110 02:51:14.902024 (+ 108us) write_op.cc:270] Start()
0110 02:51:14.902040 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469591361 usec, L: 0
0110 02:51:14.902149 (+ 109us) write_op.cc:270] Start()
0110 02:51:14.902165 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469597994 usec, L: 0
0110 02:51:14.902263 (+ 98us) write_op.cc:270] Start()
0110 02:51:14.902278 (+ 15us) write_op.cc:276] Timestamp: P: 1736477469610611 usec, L: 0
0110 02:51:14.902477 (+ 199us) write_op.cc:270] Start()
0110 02:51:14.902503 (+ 26us) write_op.cc:276] Timestamp: P: 1736477469674820 usec, L: 0
0110 02:51:14.905751 (+ 3248us) write_op.cc:270] Start()
0110 02:51:14.905775 (+ 24us) write_op.cc:276] Timestamp: P: 1736477469688450 usec, L: 0
0110 02:51:14.905981 (+ 206us) write_op.cc:270] Start()
0110 02:51:14.906007 (+ 26us) write_op.cc:276] Timestamp: P: 1736477469703071 usec, L: 0
0110 02:51:14.906147 (+ 140us) write_op.cc:270] Start()
0110 02:51:14.906169 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469724171 usec, L: 0
0110 02:51:14.906311 (+ 142us) write_op.cc:270] Start()
0110 02:51:14.906332 (+ 21us) write_op.cc:276] Timestamp: P: 1736477469733507 usec, L: 0
0110 02:51:14.906505 (+ 173us) write_op.cc:270] Start()
0110 02:51:14.906527 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469737120 usec, L: 0
0110 02:51:14.906660 (+ 133us) write_op.cc:270] Start()
0110 02:51:14.906678 (+ 18us) write_op.cc:276] Timestamp: P: 1736477469738148 usec, L: 0
0110 02:51:14.906875 (+ 197us) write_op.cc:270] Start()
0110 02:51:14.906898 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469742738 usec, L: 0
0110 02:51:14.907049 (+ 151us) write_op.cc:270] Start()
0110 02:51:14.907071 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469743534 usec, L: 0
0110 02:51:14.907210 (+ 139us) write_op.cc:270] Start()
0110 02:51:14.907231 (+ 21us) write_op.cc:276] Timestamp: P: 1736477469744470 usec, L: 0
0110 02:51:14.907366 (+ 135us) write_op.cc:270] Start()
0110 02:51:14.907389 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469750993 usec, L: 0
0110 02:51:14.907532 (+ 143us) write_op.cc:270] Start()
0110 02:51:14.907554 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469784841 usec, L: 0
0110 02:51:14.907716 (+ 162us) write_op.cc:270] Start()
0110 02:51:14.907734 (+ 18us) write_op.cc:276] Timestamp: P: 1736477469785823 usec, L: 0
0110 02:51:14.907907 (+ 173us) write_op.cc:270] Start()
0110 02:51:14.913337 (+ 5430us) write_op.cc:276] Timestamp: P: 1736477469797605 usec, L: 0
0110 02:51:14.914961 (+ 1624us) write_op.cc:270] Start()
0110 02:51:14.914989 (+ 28us) write_op.cc:276] Timestamp: P: 1736477469798332 usec, L: 0
0110 02:51:14.915140 (+ 151us) write_op.cc:270] Start()
0110 02:51:14.915159 (+ 19us) write_op.cc:276] Timestamp: P: 1736477469800039 usec, L: 0
0110 02:51:14.915302 (+ 143us) write_op.cc:270] Start()
0110 02:51:14.915324 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469804293 usec, L: 0
0110 02:51:14.915489 (+ 165us) write_op.cc:270] Start()
0110 02:51:14.915511 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469839202 usec, L: 0
0110 02:51:14.915640 (+ 129us) write_op.cc:270] Start()
0110 02:51:14.915659 (+ 19us) write_op.cc:276] Timestamp: P: 1736477469860971 usec, L: 0
0110 02:51:14.915810 (+ 151us) write_op.cc:270] Start()
0110 02:51:14.915834 (+ 24us) write_op.cc:276] Timestamp: P: 1736477469867083 usec, L: 0
0110 02:51:14.915962 (+ 128us) write_op.cc:270] Start()
0110 02:51:14.915985 (+ 23us) write_op.cc:276] Timestamp: P: 1736477469869318 usec, L: 0
0110 02:51:14.916120 (+ 135us) write_op.cc:270] Start()
0110 02:51:14.916138 (+ 18us) write_op.cc:276] Timestamp: P: 1736477469895806 usec, L: 0
0110 02:51:14.916281 (+ 143us) write_op.cc:270] Start()
0110 02:51:14.916303 (+ 22us) write_op.cc:276] Timestamp: P: 1736477469919421 usec, L: 0
0110 02:51:14.916444 (+ 141us) write_op.cc:270] Start()
0110 02:51:14.916461 (+ 17us) write_op.cc:276] Timestamp: P: 1736477469942535 usec, L: 0
0110 02:51:14.916614 (+ 153us) write_op.cc:270] Start()
0110 02:51:14.916661 (+ 47us) write_op.cc:276] Timestamp: P: 1736477469959304 usec, L: 0
0110 02:51:14.916820 (+ 159us) write_op.cc:270] Start()
0110 02:51:14.916840 (+ 20us) write_op.cc:276] Timestamp: P: 1736477469972968 usec, L: 0
0110 02:51:14.916974 (+ 134us) write_op.cc:270] Start()
0110 02:51:14.916992 (+ 18us) write_op.cc:276] Timestamp: P: 1736477469984577 usec, L: 0
0110 02:51:14.917128 (+ 136us) write_op.cc:270] Start()
0110 02:51:14.917144 (+ 16us) write_op.cc:276] Timestamp: P: 1736477469999283 usec, L: 0
0110 02:51:14.917261 (+ 117us) write_op.cc:270] Start()
0110 02:51:14.917279 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470021953 usec, L: 0
0110 02:51:14.917384 (+ 105us) write_op.cc:270] Start()
0110 02:51:14.917401 (+ 17us) write_op.cc:276] Timestamp: P: 1736477470043447 usec, L: 0
0110 02:51:14.917530 (+ 129us) write_op.cc:270] Start()
0110 02:51:14.917546 (+ 16us) write_op.cc:276] Timestamp: P: 1736477470068844 usec, L: 0
0110 02:51:14.970297 (+ 52751us) write_op.cc:270] Start()
0110 02:51:14.970333 (+ 36us) write_op.cc:276] Timestamp: P: 1736477470071157 usec, L: 0
0110 02:51:14.975996 (+ 5663us) write_op.cc:270] Start()
0110 02:51:14.976028 (+ 32us) write_op.cc:276] Timestamp: P: 1736477470072820 usec, L: 0
0110 02:51:14.976884 (+ 856us) write_op.cc:270] Start()
0110 02:51:14.976910 (+ 26us) write_op.cc:276] Timestamp: P: 1736477470098039 usec, L: 0
0110 02:51:14.977098 (+ 188us) write_op.cc:270] Start()
0110 02:51:14.977119 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470105093 usec, L: 0
0110 02:51:14.977266 (+ 147us) write_op.cc:270] Start()
0110 02:51:14.977285 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470118990 usec, L: 0
0110 02:51:14.977430 (+ 145us) write_op.cc:270] Start()
0110 02:51:14.977456 (+ 26us) write_op.cc:276] Timestamp: P: 1736477470135659 usec, L: 0
0110 02:51:14.977592 (+ 136us) write_op.cc:270] Start()
0110 02:51:14.977614 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470154686 usec, L: 0
0110 02:51:14.977747 (+ 133us) write_op.cc:270] Start()
0110 02:51:14.977763 (+ 16us) write_op.cc:276] Timestamp: P: 1736477470172496 usec, L: 0
0110 02:51:14.977878 (+ 115us) write_op.cc:270] Start()
0110 02:51:14.977902 (+ 24us) write_op.cc:276] Timestamp: P: 1736477470213920 usec, L: 0
0110 02:51:14.986910 (+ 9008us) write_op.cc:270] Start()
0110 02:51:14.986934 (+ 24us) write_op.cc:276] Timestamp: P: 1736477470237337 usec, L: 0
0110 02:51:15.000744 (+ 13810us) write_op.cc:270] Start()
0110 02:51:15.000776 (+ 32us) write_op.cc:276] Timestamp: P: 1736477470281127 usec, L: 0
0110 02:51:15.000951 (+ 175us) write_op.cc:270] Start()
0110 02:51:15.000972 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470282194 usec, L: 0
0110 02:51:15.001350 (+ 378us) write_op.cc:270] Start()
0110 02:51:15.001381 (+ 31us) write_op.cc:276] Timestamp: P: 1736477470282821 usec, L: 0
0110 02:51:15.001584 (+ 203us) write_op.cc:270] Start()
0110 02:51:15.001609 (+ 25us) write_op.cc:276] Timestamp: P: 1736477470283745 usec, L: 0
0110 02:51:15.001826 (+ 217us) write_op.cc:270] Start()
0110 02:51:15.001851 (+ 25us) write_op.cc:276] Timestamp: P: 1736477470284318 usec, L: 0
0110 02:51:15.001996 (+ 145us) write_op.cc:270] Start()
0110 02:51:15.002019 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470285538 usec, L: 0
0110 02:51:15.002163 (+ 144us) write_op.cc:270] Start()
0110 02:51:15.002210 (+ 47us) write_op.cc:276] Timestamp: P: 1736477470287428 usec, L: 0
0110 02:51:15.002410 (+ 200us) write_op.cc:270] Start()
0110 02:51:15.002446 (+ 36us) write_op.cc:276] Timestamp: P: 1736477470303915 usec, L: 0
0110 02:51:15.002570 (+ 124us) write_op.cc:270] Start()
0110 02:51:15.002591 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470305056 usec, L: 0
0110 02:51:15.002757 (+ 166us) write_op.cc:270] Start()
0110 02:51:15.002780 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470305876 usec, L: 0
0110 02:51:15.002917 (+ 137us) write_op.cc:270] Start()
0110 02:51:15.002934 (+ 17us) write_op.cc:276] Timestamp: P: 1736477470307100 usec, L: 0
0110 02:51:15.003080 (+ 146us) write_op.cc:270] Start()
0110 02:51:15.003101 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470307723 usec, L: 0
0110 02:51:15.003260 (+ 159us) write_op.cc:270] Start()
0110 02:51:15.003283 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470308290 usec, L: 0
0110 02:51:15.003422 (+ 139us) write_op.cc:270] Start()
0110 02:51:15.003444 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470308887 usec, L: 0
0110 02:51:15.003577 (+ 133us) write_op.cc:270] Start()
0110 02:51:15.003599 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470309427 usec, L: 0
0110 02:51:15.003738 (+ 139us) write_op.cc:270] Start()
0110 02:51:15.003758 (+ 20us) write_op.cc:276] Timestamp: P: 1736477470309947 usec, L: 0
0110 02:51:15.003910 (+ 152us) write_op.cc:270] Start()
0110 02:51:15.003929 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470322479 usec, L: 0
0110 02:51:15.004093 (+ 164us) write_op.cc:270] Start()
0110 02:51:15.004116 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470329027 usec, L: 0
0110 02:51:15.004263 (+ 147us) write_op.cc:270] Start()
0110 02:51:15.004285 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470352255 usec, L: 0
0110 02:51:15.004425 (+ 140us) write_op.cc:270] Start()
0110 02:51:15.004447 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470375636 usec, L: 0
0110 02:51:15.004599 (+ 152us) write_op.cc:270] Start()
0110 02:51:15.004621 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470379612 usec, L: 0
0110 02:51:15.004784 (+ 163us) write_op.cc:270] Start()
0110 02:51:15.004807 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470383053 usec, L: 0
0110 02:51:15.004947 (+ 140us) write_op.cc:270] Start()
0110 02:51:15.016417 (+ 11470us) write_op.cc:276] Timestamp: P: 1736477470384411 usec, L: 0
0110 02:51:15.016627 (+ 210us) write_op.cc:270] Start()
0110 02:51:15.016654 (+ 27us) write_op.cc:276] Timestamp: P: 1736477470416257 usec, L: 0
0110 02:51:15.016851 (+ 197us) write_op.cc:270] Start()
0110 02:51:15.016873 (+ 22us) write_op.cc:276] Timestamp: P: 1736477470444991 usec, L: 0
0110 02:51:15.017005 (+ 132us) write_op.cc:270] Start()
0110 02:51:15.017026 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470464950 usec, L: 0
0110 02:51:15.017170 (+ 144us) write_op.cc:270] Start()
0110 02:51:15.017193 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470490221 usec, L: 0
0110 02:51:15.042620 (+ 25427us) write_op.cc:270] Start()
0110 02:51:15.042645 (+ 25us) write_op.cc:276] Timestamp: P: 1736477470500117 usec, L: 0
0110 02:51:15.042794 (+ 149us) write_op.cc:270] Start()
0110 02:51:15.042813 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470500733 usec, L: 0
0110 02:51:15.042949 (+ 136us) write_op.cc:270] Start()
0110 02:51:15.042968 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470518737 usec, L: 0
0110 02:51:15.043100 (+ 132us) write_op.cc:270] Start()
0110 02:51:15.043117 (+ 17us) write_op.cc:276] Timestamp: P: 1736477470519346 usec, L: 0
0110 02:51:15.043246 (+ 129us) write_op.cc:270] Start()
0110 02:51:15.043269 (+ 23us) write_op.cc:276] Timestamp: P: 1736477470519945 usec, L: 0
0110 02:51:15.043414 (+ 145us) write_op.cc:270] Start()
0110 02:51:15.043432 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470520466 usec, L: 0
0110 02:51:15.043548 (+ 116us) write_op.cc:270] Start()
0110 02:51:15.043569 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470580466 usec, L: 0
0110 02:51:15.043714 (+ 145us) write_op.cc:270] Start()
0110 02:51:15.043734 (+ 20us) write_op.cc:276] Timestamp: P: 1736477470581579 usec, L: 0
0110 02:51:15.043867 (+ 133us) write_op.cc:270] Start()
0110 02:51:15.043888 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470582153 usec, L: 0
0110 02:51:15.044009 (+ 121us) write_op.cc:270] Start()
0110 02:51:15.044028 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470582733 usec, L: 0
0110 02:51:15.044150 (+ 122us) write_op.cc:270] Start()
0110 02:51:15.044168 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470583411 usec, L: 0
0110 02:51:15.044274 (+ 106us) write_op.cc:270] Start()
0110 02:51:15.044302 (+ 28us) write_op.cc:276] Timestamp: P: 1736477470595870 usec, L: 0
0110 02:51:15.044432 (+ 130us) write_op.cc:270] Start()
0110 02:51:15.044450 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470598062 usec, L: 0
0110 02:51:15.044567 (+ 117us) write_op.cc:270] Start()
0110 02:51:15.044583 (+ 16us) write_op.cc:276] Timestamp: P: 1736477470598927 usec, L: 0
0110 02:51:15.044700 (+ 117us) write_op.cc:270] Start()
0110 02:51:15.044718 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470607567 usec, L: 0
0110 02:51:15.045416 (+ 698us) write_op.cc:270] Start()
0110 02:51:15.045470 (+ 54us) write_op.cc:276] Timestamp: P: 1736477470608470 usec, L: 0
0110 02:51:15.066844 (+ 21374us) write_op.cc:270] Start()
0110 02:51:15.066885 (+ 41us) write_op.cc:276] Timestamp: P: 1736477470772578 usec, L: 0
0110 02:51:15.067120 (+ 235us) write_op.cc:270] Start()
0110 02:51:15.067148 (+ 28us) write_op.cc:276] Timestamp: P: 1736477470773097 usec, L: 0
0110 02:51:15.067327 (+ 179us) write_op.cc:270] Start()
0110 02:51:15.067352 (+ 25us) write_op.cc:276] Timestamp: P: 1736477470777135 usec, L: 0
0110 02:51:15.067495 (+ 143us) write_op.cc:270] Start()
0110 02:51:15.067515 (+ 20us) write_op.cc:276] Timestamp: P: 1736477470779439 usec, L: 0
0110 02:51:15.067860 (+ 345us) write_op.cc:270] Start()
0110 02:51:15.067907 (+ 47us) write_op.cc:276] Timestamp: P: 1736477470795173 usec, L: 0
0110 02:51:15.068067 (+ 160us) write_op.cc:270] Start()
0110 02:51:15.068086 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470795963 usec, L: 0
0110 02:51:15.068205 (+ 119us) write_op.cc:270] Start()
0110 02:51:15.068224 (+ 19us) write_op.cc:276] Timestamp: P: 1736477470805013 usec, L: 0
0110 02:51:15.068343 (+ 119us) write_op.cc:270] Start()
0110 02:51:15.068361 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470805661 usec, L: 0
0110 02:51:15.078324 (+ 9963us) write_op.cc:270] Start()
0110 02:51:15.078349 (+ 25us) write_op.cc:276] Timestamp: P: 1736477470820240 usec, L: 0
0110 02:51:15.078516 (+ 167us) write_op.cc:270] Start()
0110 02:51:15.078561 (+ 45us) write_op.cc:276] Timestamp: P: 1736477470843419 usec, L: 0
0110 02:51:15.078742 (+ 181us) write_op.cc:270] Start()
0110 02:51:15.078762 (+ 20us) write_op.cc:276] Timestamp: P: 1736477470844256 usec, L: 0
0110 02:51:15.078893 (+ 131us) write_op.cc:270] Start()
0110 02:51:15.078911 (+ 18us) write_op.cc:276] Timestamp: P: 1736477470845163 usec, L: 0
0110 02:51:15.079027 (+ 116us) write_op.cc:270] Start()
0110 02:51:15.079044 (+ 17us) write_op.cc:276] Timestamp: P: 1736477470908743 usec, L: 0
0110 02:51:15.079169 (+ 125us) write_op.cc:270] Start()
0110 02:51:15.079190 (+ 21us) write_op.cc:276] Timestamp: P: 1736477470963041 usec, L: 0
0110 02:51:15.079307 (+ 117us) write_op.cc:270] Start()
0110 02:51:15.079324 (+ 17us) write_op.cc:276] Timestamp: P: 1736477471014445 usec, L: 0
0110 02:51:15.079456 (+ 132us) write_op.cc:270] Start()
0110 02:51:15.079473 (+ 17us) write_op.cc:276] Timestamp: P: 1736477471016065 usec, L: 0
0110 02:51:15.079587 (+ 114us) write_op.cc:270] Start()
0110 02:51:15.079604 (+ 17us) write_op.cc:276] Timestamp: P: 1736477471017275 usec, L: 0
0110 02:51:15.079726 (+ 122us) write_op.cc:270] Start()
0110 02:51:15.079743 (+ 17us) write_op.cc:276] Timestamp: P: 1736477471018411 usec, L: 0
0110 02:51:15.079853 (+ 110us) write_op.cc:270] Start()
0110 02:51:15.079874 (+ 21us) write_op.cc:276] Timestamp: P: 1736477471019611 usec, L: 0
0110 02:51:15.090180 (+ 10306us) write_op.cc:270] Start()
0110 02:51:15.090238 (+ 58us) write_op.cc:276] Timestamp: P: 1736477471044440 usec, L: 0
0110 02:51:15.090956 (+ 718us) write_op.cc:270] Start()
0110 02:51:15.091080 (+ 124us) write_op.cc:276] Timestamp: P: 1736477471074773 usec, L: 0
0110 02:51:15.091840 (+ 760us) write_op.cc:270] Start()
0110 02:51:15.091972 (+ 132us) write_op.cc:276] Timestamp: P: 1736477471152811 usec, L: 0
0110 02:51:15.092493 (+ 521us) write_op.cc:270] Start()
0110 02:51:15.092610 (+ 117us) write_op.cc:276] Timestamp: P: 1736477471257860 usec, L: 0
0110 02:51:15.093155 (+ 545us) write_op.cc:270] Start()
0110 02:51:15.093274 (+ 119us) write_op.cc:276] Timestamp: P: 1736477471524257 usec, L: 0
0110 02:51:15.093944 (+ 670us) write_op.cc:270] Start()
0110 02:51:15.094434 (+ 490us) write_op.cc:276] Timestamp: P: 1736477471526162 usec, L: 0
0110 02:51:15.094880 (+ 446us) write_op.cc:270] Start()
0110 02:51:15.095074 (+ 194us) write_op.cc:276] Timestamp: P: 1736477471527050 usec, L: 0
0110 02:51:15.095662 (+ 588us) write_op.cc:270] Start()
0110 02:51:15.095784 (+ 122us) write_op.cc:276] Timestamp: P: 1736477471555765 usec, L: 0
0110 02:51:15.096309 (+ 525us) write_op.cc:270] Start()
0110 02:51:15.096417 (+ 108us) write_op.cc:276] Timestamp: P: 1736477471557630 usec, L: 0
0110 02:51:15.096901 (+ 484us) write_op.cc:270] Start()
0110 02:51:15.096927 (+ 26us) write_op.cc:276] Timestamp: P: 1736477471558732 usec, L: 0
0110 02:51:15.097560 (+ 633us) write_op.cc:270] Start()
0110 02:51:15.097671 (+ 111us) write_op.cc:276] Timestamp: P: 1736477
W20250110 02:51:16.236186 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e554d00d05b4fa7a14c9b6159f9d726 consensus state has no leader
I20250110 02:51:16.236599 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_cba45bffb2a3419c83fafeea7087a9f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.237169 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba45bffb2a3419c83fafeea7087a9f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.238775 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3b60f335e3574a5bbf7ab20a93a64bed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.239218 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b60f335e3574a5bbf7ab20a93a64bed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:16.240979 524 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54b538100977456794c9519bbdffa7e7 consensus state has no leader
I20250110 02:51:16.241350 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7cebf0b5d59d4f068fc8dce07001bd12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.241868 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cebf0b5d59d4f068fc8dce07001bd12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:16.242630 526 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e0b2ec1d5814144bc58261e6c6e8d35 consensus state has no leader
I20250110 02:51:16.242988 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a993dbbdb6714184867df68d17523ca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.243433 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a993dbbdb6714184867df68d17523ca7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.263319 1184 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.263516 1184 ts_tablet_manager.cc:1397] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.171s user 0.062s sys 0.040s
I20250110 02:51:16.265712 1184 raft_consensus.cc:357] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.265995 1184 raft_consensus.cc:383] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.266624 1184 raft_consensus.cc:738] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.266975 1184 consensus_queue.cc:260] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.267558 1184 ts_tablet_manager.cc:1428] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:16.268644 1181 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.268859 1181 ts_tablet_manager.cc:1397] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.010s
I20250110 02:51:16.269073 1184 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.269747 1181 raft_consensus.cc:357] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.269999 1181 raft_consensus.cc:383] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.270121 1181 raft_consensus.cc:738] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.270435 1181 consensus_queue.cc:260] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.279596 1184 tablet_bootstrap.cc:654] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.298218 1682 consensus_queue.cc:1035] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:16.310500 1178 ts_tablet_manager.cc:1918] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.310742 1178 log.cc:1198] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/514704cd935b44e8b760221224332509
I20250110 02:51:16.311348 1178 ts_tablet_manager.cc:1939] T 514704cd935b44e8b760221224332509 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:16.312172 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 514704cd935b44e8b760221224332509 (table default.Table_7377066199724b26af78c183f9811324 [id=5ac50ad7765242968bb9f7d7986e9207]) successfully deleted
I20250110 02:51:16.315605 1605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8bc934cc708045e9858ee8f8e64660cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:16.315884 1605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: LEADER
W20250110 02:51:16.323683 1673 log.cc:926] Time spent T 1cc0532c94eb401e9d299e79556b4cf1 P 85f9597077934f1a9814d284ed06d13d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250110 02:51:16.324944 1188 tablet_bootstrap.cc:492] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:16.325110 1188 ts_tablet_manager.cc:1397] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.409s user 0.005s sys 0.030s
I20250110 02:51:16.325865 1188 raft_consensus.cc:357] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.326108 1188 raft_consensus.cc:383] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.326200 1188 raft_consensus.cc:738] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.326427 1188 consensus_queue.cc:260] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.327157 1188 ts_tablet_manager.cc:1428] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:16.327440 1188 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:16.328991 1188 tablet_bootstrap.cc:654] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.332031 1178 tablet_replica.cc:331] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:16.334199 1178 raft_consensus.cc:2238] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.334612 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_8d7fe23e0afe4dfdac17778b492f2c55 [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 3 got 0
I20250110 02:51:16.343483 523 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:16.345731 523 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:16.346568 1178 raft_consensus.cc:2267] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.347404 1178 ts_tablet_manager.cc:1905] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.355862 1039 tablet_service.cc:1467] Processing CreateTablet for tablet c28f191afa2a43608bbd74322bec97a6 (DEFAULT_TABLE table=default.Table_82157dfbcfe64e759c6847722284ee3f [id=c6f818c8ec264520a7f85dd60ab6ce1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.367197 1693 tablet.cc:1721] T 00618f611d4c4a84a988125c5d175efc P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:16.367687 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28f191afa2a43608bbd74322bec97a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.346256 889 tablet_service.cc:1467] Processing CreateTablet for tablet c28f191afa2a43608bbd74322bec97a6 (DEFAULT_TABLE table=default.Table_82157dfbcfe64e759c6847722284ee3f [id=c6f818c8ec264520a7f85dd60ab6ce1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.371701 756 tablet_service.cc:1467] Processing CreateTablet for tablet c28f191afa2a43608bbd74322bec97a6 (DEFAULT_TABLE table=default.Table_82157dfbcfe64e759c6847722284ee3f [id=c6f818c8ec264520a7f85dd60ab6ce1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.372201 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28f191afa2a43608bbd74322bec97a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.381698 1689 tablet.cc:1721] T 00618f611d4c4a84a988125c5d175efc P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:16.386298 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_8d7fe23e0afe4dfdac17778b492f2c55 [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 3 got 0
I20250110 02:51:16.387506 1039 tablet_service.cc:1467] Processing CreateTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 (DEFAULT_TABLE table=default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.388036 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ac15f4e774b7e854e4c5b4737b591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.388736 1678 raft_consensus.cc:491] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.388960 1678 raft_consensus.cc:513] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.390462 1678 leader_election.cc:290] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:16.390498 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:16.390820 775 raft_consensus.cc:2463] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:16.391232 996 leader_election.cc:304] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:16.391552 1678 raft_consensus.cc:2798] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:16.391732 1678 raft_consensus.cc:491] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:16.391881 1678 raft_consensus.cc:3054] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.394496 1678 raft_consensus.cc:513] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.395579 1678 leader_election.cc:290] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:16.396006 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:16.396339 775 raft_consensus.cc:3054] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.399680 775 raft_consensus.cc:2463] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:16.400185 996 leader_election.cc:304] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:16.400535 1678 raft_consensus.cc:2798] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:16.400827 1678 raft_consensus.cc:695] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:16.401126 1678 consensus_queue.cc:237] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.405575 754 tablet_service.cc:1467] Processing CreateTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 (DEFAULT_TABLE table=default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.406041 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ac15f4e774b7e854e4c5b4737b591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.417851 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e (DEFAULT_TABLE table=default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.418337 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9b3e74ee2431a85a22fefaacfa40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.435777 1588 raft_consensus.cc:491] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.435981 1588 raft_consensus.cc:513] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.436606 1588 leader_election.cc:290] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:16.437539 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:16.438436 711 leader_election.cc:343] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.439656 754 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (DEFAULT_TABLE table=default.Table_61c4e10750aa4566916853b2c35ed9fb [id=0a7a61acdae94ee392948eaa9495dfcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.440953 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf81039d5e4a47b426ea5d2f21acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.460350 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:16.460531 1184 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.460680 1184 ts_tablet_manager.cc:1397] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.192s user 0.020s sys 0.019s
W20250110 02:51:16.461314 994 leader_election.cc:343] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.461470 1184 raft_consensus.cc:357] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.461741 1184 raft_consensus.cc:383] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.461853 1184 raft_consensus.cc:738] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.462076 1184 consensus_queue.cc:260] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.462800 1184 ts_tablet_manager.cc:1428] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:16.463112 1184 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.463603 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.464453 709 leader_election.cc:343] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.464601 709 leader_election.cc:304] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:16.465114 884 tablet_service.cc:1467] Processing CreateTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e (DEFAULT_TABLE table=default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.465456 883 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (DEFAULT_TABLE table=default.Table_61c4e10750aa4566916853b2c35ed9fb [id=0a7a61acdae94ee392948eaa9495dfcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.466132 1690 tablet.cc:1721] T 00618f611d4c4a84a988125c5d175efc P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:16.466444 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.467096 994 leader_election.cc:343] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.467519 892 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9f920cde954f31acebc6d0b6965e58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.467790 898 tablet_service.cc:1467] Processing CreateTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 (DEFAULT_TABLE table=default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.468163 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c28f191afa2a43608bbd74322bec97a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.468525 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf81039d5e4a47b426ea5d2f21acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.473995 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9b3e74ee2431a85a22fefaacfa40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.484017 1588 raft_consensus.cc:2743] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:16.484056 1184 tablet_bootstrap.cc:654] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.484496 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5ac15f4e774b7e854e4c5b4737b591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.495637 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 818dcbf83303412d97b097913bcd36c3 (DEFAULT_TABLE table=default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.496197 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818dcbf83303412d97b097913bcd36c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.499615 1184 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.500366 1184 ts_tablet_manager.cc:1397] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.005s
I20250110 02:51:16.501261 1184 raft_consensus.cc:357] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:16.447530 510 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (table default.Table_c2f177ce9d6a420bbc77b33929ccaa7a [id=f18e7a9e2e274c2d86fb5d9455ecc4fe], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e0b2ec1d5814144bc58261e6c6e8d35 consensus state has no leader
I20250110 02:51:16.501510 1184 raft_consensus.cc:383] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.501581 1184 raft_consensus.cc:738] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.501849 1184 consensus_queue.cc:260] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.484864 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5156f6b732b14eb6807a520bdfa15b62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:16.502396 1184 ts_tablet_manager.cc:1428] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:16.502758 1184 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:16.502758 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5156f6b732b14eb6807a520bdfa15b62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.504245 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:16.506595 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.506757 884 tablet_service.cc:1467] Processing CreateTablet for tablet 818dcbf83303412d97b097913bcd36c3 (DEFAULT_TABLE table=default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.507133 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818dcbf83303412d97b097913bcd36c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:16.507531 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 89f6797b1076468a93fb0b510c553bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f6797b1076468a93fb0b510c553bed already in progress: creating tablet
I20250110 02:51:16.509208 1184 tablet_bootstrap.cc:654] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.511193 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.511525 1180 tablet_replica.cc:331] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.511786 1180 raft_consensus.cc:2238] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.511993 1180 raft_consensus.cc:2267] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.512917 1588 raft_consensus.cc:491] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.513144 1180 ts_tablet_manager.cc:1905] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.513077 1588 raft_consensus.cc:513] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.513697 1588 leader_election.cc:290] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:16.514504 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:16.514835 756 tablet_service.cc:1467] Processing CreateTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e (DEFAULT_TABLE table=default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:16.515132 711 leader_election.cc:343] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.515380 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9b3e74ee2431a85a22fefaacfa40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.514531 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (DEFAULT_TABLE table=default.Table_61c4e10750aa4566916853b2c35ed9fb [id=0a7a61acdae94ee392948eaa9495dfcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.515657 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:16.515895 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cf81039d5e4a47b426ea5d2f21acc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:16.518697 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 58440.5us
I20250110 02:51:16.519379 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 302af84fd8374ffa96a7503b54849536) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:16.519835 709 leader_election.cc:343] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.520062 709 leader_election.cc:304] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:16.520915 1588 raft_consensus.cc:2743] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:16.524945 1180 ts_tablet_manager.cc:1918] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.525190 1180 log.cc:1198] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:16.525928 1180 ts_tablet_manager.cc:1939] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.527247 753 tablet_service.cc:1467] Processing CreateTablet for tablet 818dcbf83303412d97b097913bcd36c3 (DEFAULT_TABLE table=default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.527758 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818dcbf83303412d97b097913bcd36c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.535838 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 (DEFAULT_TABLE table=default.Table_7431ea53e3704aebb4d79e0a0c4045f5 [id=70ad733a8a4943a39c52ee59ad2253cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.536209 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55edfa05075b4edcb70ef29e5ee413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.537470 1184 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.537607 1184 ts_tablet_manager.cc:1397] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20250110 02:51:16.538424 1184 raft_consensus.cc:357] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.538674 1184 raft_consensus.cc:383] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.538763 1184 raft_consensus.cc:738] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.538975 1184 consensus_queue.cc:260] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.539526 1184 ts_tablet_manager.cc:1428] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:16.539459 898 tablet_service.cc:1467] Processing CreateTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 (DEFAULT_TABLE table=default.Table_7431ea53e3704aebb4d79e0a0c4045f5 [id=70ad733a8a4943a39c52ee59ad2253cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.539829 1184 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.539894 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55edfa05075b4edcb70ef29e5ee413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.541546 1184 tablet_bootstrap.cc:654] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.545275 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4e54ae5d7e864aff974a107f6762f446 (DEFAULT_TABLE table=default.Table_febf1d5b7a274c5fb59e10a180b4dd59 [id=23e8c0cf343748a092d750ea2ff7647a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.545675 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e54ae5d7e864aff974a107f6762f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.549312 1181 ts_tablet_manager.cc:1428] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.280s user 0.012s sys 0.026s
I20250110 02:51:16.550349 753 tablet_service.cc:1467] Processing CreateTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 (DEFAULT_TABLE table=default.Table_7431ea53e3704aebb4d79e0a0c4045f5 [id=70ad733a8a4943a39c52ee59ad2253cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.550930 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55edfa05075b4edcb70ef29e5ee413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.551539 1181 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.559674 1188 tablet_bootstrap.cc:492] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:16.559919 1188 ts_tablet_manager.cc:1397] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.233s user 0.004s sys 0.030s
I20250110 02:51:16.560946 1188 raft_consensus.cc:357] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.561275 1188 raft_consensus.cc:383] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.561421 1188 raft_consensus.cc:738] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.561779 1188 consensus_queue.cc:260] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.563932 1188 ts_tablet_manager.cc:1428] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:16.567613 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 4e54ae5d7e864aff974a107f6762f446 (DEFAULT_TABLE table=default.Table_febf1d5b7a274c5fb59e10a180b4dd59 [id=23e8c0cf343748a092d750ea2ff7647a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.568483 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e54ae5d7e864aff974a107f6762f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.575650 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (table default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]) successfully deleted
I20250110 02:51:16.576086 1180 tablet_replica.cc:331] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:16.576360 1180 raft_consensus.cc:2238] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.576557 1180 raft_consensus.cc:2267] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.577818 1180 ts_tablet_manager.cc:1905] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.580127 740 tablet_service.cc:1467] Processing CreateTablet for tablet 4e54ae5d7e864aff974a107f6762f446 (DEFAULT_TABLE table=default.Table_febf1d5b7a274c5fb59e10a180b4dd59 [id=23e8c0cf343748a092d750ea2ff7647a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.580540 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e54ae5d7e864aff974a107f6762f446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.583110 892 tablet_service.cc:1467] Processing CreateTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.583657 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071bbce1706c43b6bb690c7de1ead3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.584254 1188 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:16.586643 1181 tablet_bootstrap.cc:654] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.592686 1178 ts_tablet_manager.cc:1918] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.592820 1178 log.cc:1198] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1dfc062b5fcb42c3bce6dfaed54b3869
I20250110 02:51:16.593214 1178 ts_tablet_manager.cc:1939] T 1dfc062b5fcb42c3bce6dfaed54b3869 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:16.594324 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1dfc062b5fcb42c3bce6dfaed54b3869 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
I20250110 02:51:16.594738 1178 tablet_replica.cc:331] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:16.594689 741 tablet_service.cc:1467] Processing CreateTablet for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4 (DEFAULT_TABLE table=default.Table_bb1c63b5d7ee453a999514a777e71496 [id=ab05074705fa48438c82fd089d97be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.594938 1178 raft_consensus.cc:2238] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.595078 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.595121 1178 raft_consensus.cc:2267] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.597129 1188 tablet_bootstrap.cc:654] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.602700 744 tablet_service.cc:1467] Processing CreateTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.603060 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071bbce1706c43b6bb690c7de1ead3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.603855 1039 tablet_service.cc:1467] Processing CreateTablet for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4 (DEFAULT_TABLE table=default.Table_bb1c63b5d7ee453a999514a777e71496 [id=ab05074705fa48438c82fd089d97be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.604233 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.605906 519 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:16.606148 519 catalog_manager.cc:5526] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:16.606676 519 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:16.610498 1680 raft_consensus.cc:491] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.610641 1680 raft_consensus.cc:513] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.611418 1680 leader_election.cc:290] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:16.612015 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:16.612308 774 raft_consensus.cc:2463] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:16.612880 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_fe9783d91cae4578b6f0e7a6cc61bc84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:16.613356 996 leader_election.cc:304] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:16.613502 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.613718 1680 raft_consensus.cc:2798] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:16.613865 1680 raft_consensus.cc:491] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:16.613991 1680 raft_consensus.cc:3054] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.614759 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:16.615496 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_3f51e7f8aaa7470fbeeb5459695efab1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.615940 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f51e7f8aaa7470fbeeb5459695efab1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.618054 1680 raft_consensus.cc:513] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.618906 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.619017 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.619644 994 leader_election.cc:343] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:16.619877 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1390f867a3a8436092d1920daff4c5b0 already in progress: creating tablet
I20250110 02:51:16.621325 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:16.621585 774 raft_consensus.cc:3054] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.623975 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:16.624452 522 master_service.cc:946] ReplaceTablet: received request to replace tablet 8eb9ead3175140028e584e1ccb4874de from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:16.625223 774 raft_consensus.cc:2463] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:16.625752 996 leader_election.cc:304] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:16.626118 1678 raft_consensus.cc:2798] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:16.626291 1678 raft_consensus.cc:695] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:16.626582 1678 consensus_queue.cc:237] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.627413 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.628037 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:16.628599 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4af23720048fe98b3231ba402d45c already in progress: creating tablet
I20250110 02:51:16.629292 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:16.630290 994 leader_election.cc:343] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.630439 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.631240 1680 leader_election.cc:290] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:16.633014 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 55edfa05075b4edcb70ef29e5ee413c6
I20250110 02:51:16.633704 520 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d5ddbc6b02742609bc2ca84c863a9d0 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:16.635146 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.635645 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.635700 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:16.636291 512 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because the tablet was not found. No further retry: Not found: Tablet not found: 55edfa05075b4edcb70ef29e5ee413c6
I20250110 02:51:16.637341 1181 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.637529 1181 ts_tablet_manager.cc:1397] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.004s
I20250110 02:51:16.637624 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_31a8cb16aea94ae7ae6370b386b9dd82" } new_table_name: "default.Table_72a27bbc01334e52ae01e2c46eab0581" modify_external_catalogs: true
I20250110 02:51:16.638581 1181 raft_consensus.cc:357] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.638824 1181 raft_consensus.cc:383] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.638927 1181 raft_consensus.cc:738] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.643949 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:16.644358 1181 consensus_queue.cc:260] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.645494 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_78b575ae2603466d9703e8780d3cd413" } new_table_name: "default.Table_c4a5fd4dc8a94e108deada6468c9fa8e" modify_external_catalogs: true
I20250110 02:51:16.645645 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.645874 740 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:16.646420 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa6057de9da04f2c82b8f25c5b4e2fe4 already in progress: creating tablet
I20250110 02:51:16.647388 1181 ts_tablet_manager.cc:1428] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250110 02:51:16.647739 1181 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.647969 1184 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.648129 1184 ts_tablet_manager.cc:1397] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.025s
I20250110 02:51:16.648648 892 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.648898 1184 raft_consensus.cc:357] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.649112 1184 raft_consensus.cc:383] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.649188 1184 raft_consensus.cc:738] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.649438 1184 consensus_queue.cc:260] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.649974 1184 ts_tablet_manager.cc:1428] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:16.650272 1184 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.652458 1184 tablet_bootstrap.cc:654] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.655309 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet c28f191afa2a43608bbd74322bec97a6
I20250110 02:51:16.656180 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_9cd2f808ffbc4122be3501fc1a25c768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.656785 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd2f808ffbc4122be3501fc1a25c768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.658723 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.658994 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.659106 1178 ts_tablet_manager.cc:1905] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.659546 898 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
W20250110 02:51:16.659852 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:16.660121 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 452223af5f6a4836a0f582d394a9453e from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:16.660789 1184 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.660952 1184 ts_tablet_manager.cc:1397] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20250110 02:51:16.653630 1188 tablet_bootstrap.cc:492] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:16.661830 1188 ts_tablet_manager.cc:1397] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.000s
I20250110 02:51:16.661818 1184 raft_consensus.cc:357] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.661986 1184 raft_consensus.cc:383] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.662075 1184 raft_consensus.cc:738] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.662976 1188 raft_consensus.cc:357] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.668861 1188 raft_consensus.cc:383] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.669436 1181 tablet_bootstrap.cc:654] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.669595 1188 raft_consensus.cc:738] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.668507 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.670811 1188 consensus_queue.cc:260] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.672855 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 248055ca83de4e62a6b7e908c53d3d9a (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.672969 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1df15c10ef2d4bf99af06a860c6f250f (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.673247 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248055ca83de4e62a6b7e908c53d3d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.673206 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe (DEFAULT_TABLE table=default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.673466 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df15c10ef2d4bf99af06a860c6f250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.673664 740 tablet_service.cc:1467] Processing CreateTablet for tablet 248055ca83de4e62a6b7e908c53d3d9a (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.674074 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248055ca83de4e62a6b7e908c53d3d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.674775 1188 ts_tablet_manager.cc:1428] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250110 02:51:16.675069 1188 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:16.676451 1180 ts_tablet_manager.cc:1918] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.676580 1180 log.cc:1198] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1602ade49b7442d6b8e6f4b85d2e80a1
I20250110 02:51:16.677059 1180 ts_tablet_manager.cc:1939] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:16.678754 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1df15c10ef2d4bf99af06a860c6f250f (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.679086 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df15c10ef2d4bf99af06a860c6f250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.672279 898 tablet_service.cc:1467] Processing CreateTablet for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (DEFAULT_TABLE table=default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.681758 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.682716 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071bbce1706c43b6bb690c7de1ead3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.683845 1188 tablet_bootstrap.cc:654] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.684942 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1df15c10ef2d4bf99af06a860c6f250f (DEFAULT_TABLE table=default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.685317 884 tablet_service.cc:1467] Processing CreateTablet for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4 (DEFAULT_TABLE table=default.Table_bb1c63b5d7ee453a999514a777e71496 [id=ab05074705fa48438c82fd089d97be75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.686599 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7064f3f16f6144a4b624eff2ad3ccf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.686928 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1602ade49b7442d6b8e6f4b85d2e80a1 (table default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]) successfully deleted
I20250110 02:51:16.690501 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.694541 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df15c10ef2d4bf99af06a860c6f250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.696146 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8a0645a517224b7ea46c9901cb84ef08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.696631 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a0645a517224b7ea46c9901cb84ef08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.697957 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_1d2e132b16324d3ea79f29a7339f4d5e" } new_table_name: "default.Table_94b5e8f76ac043d3b234eb6f8b625e2f" modify_external_catalogs: true
I20250110 02:51:16.698617 522 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:16.698994 522 catalog_manager.cc:6596] ReplaceTablet: tablet 8eb9ead3175140028e584e1ccb4874de deleted and replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e
I20250110 02:51:16.699313 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_35002d6ab8564fe1b93188d7ce973d3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.699715 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35002d6ab8564fe1b93188d7ce973d3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:16.702021 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6757b7ecc514489a0fa7e1ff18206bd consensus state has no leader
I20250110 02:51:16.702144 883 tablet_service.cc:1467] Processing CreateTablet for tablet 248055ca83de4e62a6b7e908c53d3d9a (DEFAULT_TABLE table=default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.702350 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_101d4ba71c6d4b7b9dc84b11a5dbd65a" } new_table_name: "default.Table_8de67d45a9a54567afbcd1896b321c97" modify_external_catalogs: true
I20250110 02:51:16.702491 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248055ca83de4e62a6b7e908c53d3d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.703204 1181 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.703354 1181 ts_tablet_manager.cc:1397] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20250110 02:51:16.703939 882 tablet_service.cc:1467] Processing CreateTablet for tablet 07c4c653a5954d2789abd1611e3c64d0 (DEFAULT_TABLE table=default.Table_34eb9282d67443a2849ec525cdaa9e7b [id=f73acd74cc49421ea2e999c9de28b4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.704012 1181 raft_consensus.cc:357] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.704224 1181 raft_consensus.cc:383] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.704269 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c4c653a5954d2789abd1611e3c64d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.704290 1181 raft_consensus.cc:738] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.704522 1181 consensus_queue.cc:260] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.706071 887 tablet_service.cc:1467] Processing CreateTablet for tablet a0ddab9e847445999ec97eac4352d9c2 (DEFAULT_TABLE table=default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.706391 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ddab9e847445999ec97eac4352d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.706375 1181 ts_tablet_manager.cc:1428] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:16.706689 1181 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.708021 893 tablet_service.cc:1467] Processing CreateTablet for tablet b48fe2d061da4e729c8704af4fc83f7a (DEFAULT_TABLE table=default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 [id=2fe86361000e43eda29c3a39e78beeee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.708317 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fe2d061da4e729c8704af4fc83f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.710799 1181 tablet_bootstrap.cc:654] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:16.714560 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4af23720048fe98b3231ba402d45c already in progress: creating tablet
I20250110 02:51:16.715581 1024 tablet_service.cc:1467] Processing CreateTablet for tablet a0ddab9e847445999ec97eac4352d9c2 (DEFAULT_TABLE table=default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.715709 1178 ts_tablet_manager.cc:1918] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.715807 1178 log.cc:1198] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/ad9f920cde954f31acebc6d0b6965e58
I20250110 02:51:16.715934 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ddab9e847445999ec97eac4352d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.716131 1178 ts_tablet_manager.cc:1939] T ad9f920cde954f31acebc6d0b6965e58 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:16.716216 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:16.716631 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
W20250110 02:51:16.716876 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa6057de9da04f2c82b8f25c5b4e2fe4 already in progress: creating tablet
W20250110 02:51:16.717106 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:16.717036 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 07c4c653a5954d2789abd1611e3c64d0 (DEFAULT_TABLE table=default.Table_34eb9282d67443a2849ec525cdaa9e7b [id=f73acd74cc49421ea2e999c9de28b4ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:16.717280 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1390f867a3a8436092d1920daff4c5b0 already in progress: creating tablet
W20250110 02:51:16.717461 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9ead3175140028e584e1ccb4874de already in progress: creating tablet
I20250110 02:51:16.717522 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet ad9f920cde954f31acebc6d0b6965e58 (table default.Table_3edf95be0d264f409133f83c3e948d58 [id=3a48ad78300a42c691144ec0fac7e788]) successfully deleted
I20250110 02:51:16.717587 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c4c653a5954d2789abd1611e3c64d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:16.718164 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4af23720048fe98b3231ba402d45c already in progress: creating tablet
I20250110 02:51:16.718698 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b48fe2d061da4e729c8704af4fc83f7a (DEFAULT_TABLE table=default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 [id=2fe86361000e43eda29c3a39e78beeee]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:16.718784 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1390f867a3a8436092d1920daff4c5b0 already in progress: creating tablet
I20250110 02:51:16.718861 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.719099 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fe2d061da4e729c8704af4fc83f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.719260 744 tablet_service.cc:1467] Processing CreateTablet for tablet b48fe2d061da4e729c8704af4fc83f7a (DEFAULT_TABLE table=default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 [id=2fe86361000e43eda29c3a39e78beeee]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:16.719365 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa6057de9da04f2c82b8f25c5b4e2fe4 already in progress: creating tablet
I20250110 02:51:16.719632 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48fe2d061da4e729c8704af4fc83f7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:16.720078 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
W20250110 02:51:16.720556 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
I20250110 02:51:16.722715 740 tablet_service.cc:1467] Processing CreateTablet for tablet a0ddab9e847445999ec97eac4352d9c2 (DEFAULT_TABLE table=default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.723042 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0ddab9e847445999ec97eac4352d9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.724340 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (DEFAULT_TABLE table=default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.724411 523 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:16.724712 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7064f3f16f6144a4b624eff2ad3ccf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.727499 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_2e282311da134ac6abccffab8398357b" } new_table_name: "default.Table_f752952b67ed41d48e3e639949d6869a" modify_external_catalogs: true
I20250110 02:51:16.730783 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:16.731388 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9ead3175140028e584e1ccb4874de already in progress: creating tablet
I20250110 02:51:16.731909 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (DEFAULT_TABLE table=default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.732378 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7064f3f16f6144a4b624eff2ad3ccf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.662325 1184 consensus_queue.cc:260] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.738205 1184 ts_tablet_manager.cc:1428] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.077s user 0.001s sys 0.000s
W20250110 02:51:16.739781 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb9ead3175140028e584e1ccb4874de already in progress: creating tablet
I20250110 02:51:16.740406 1184 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.742497 753 tablet_service.cc:1467] Processing CreateTablet for tablet 07c4c653a5954d2789abd1611e3c64d0 (DEFAULT_TABLE table=default.Table_34eb9282d67443a2849ec525cdaa9e7b [id=f73acd74cc49421ea2e999c9de28b4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:16.744347 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c4c653a5954d2789abd1611e3c64d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:16.745810 1184 tablet_bootstrap.cc:654] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:16.752002 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39cfc508c1b84299a2f16d307bb25354 consensus state has no leader
I20250110 02:51:16.754621 1188 tablet_bootstrap.cc:492] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:16.754791 1188 ts_tablet_manager.cc:1397] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.001s
I20250110 02:51:16.755622 1188 raft_consensus.cc:357] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.755851 1188 raft_consensus.cc:383] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.755936 1188 raft_consensus.cc:738] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.756155 1188 consensus_queue.cc:260] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.756670 1188 ts_tablet_manager.cc:1428] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:16.756974 1188 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:16.761250 1188 tablet_bootstrap.cc:654] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.766036 774 raft_consensus.cc:1270] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Refusing update from remote peer 8899dd1aeb384e4ea4523f5fd225c888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:16.766227 1181 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.766407 1181 ts_tablet_manager.cc:1397] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.001s
W20250110 02:51:16.766867 994 consensus_peers.cc:487] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:16.767356 1680 consensus_queue.cc:1035] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:16.772454 1184 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.779225 1184 ts_tablet_manager.cc:1397] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.004s
I20250110 02:51:16.783946 1181 raft_consensus.cc:357] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.784200 1181 raft_consensus.cc:383] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.784279 1181 raft_consensus.cc:738] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.784547 1181 consensus_queue.cc:260] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.785295 1181 ts_tablet_manager.cc:1428] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250110 02:51:16.786001 517 master_service.cc:946] ReplaceTablet: received request to replace tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:16.787516 1685 raft_consensus.cc:491] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.787694 1685 raft_consensus.cc:513] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.788514 1685 leader_election.cc:290] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:16.790230 853 leader_election.cc:343] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:16.780498 1184 raft_consensus.cc:357] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.794672 1184 raft_consensus.cc:383] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.794955 1184 raft_consensus.cc:738] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.795712 1184 consensus_queue.cc:260] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.796168 1689 tablet.cc:1721] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:16.798336 1184 ts_tablet_manager.cc:1428] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250110 02:51:16.800395 1184 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:16.803404 1693 tablet.cc:1721] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:16.805286 1184 tablet_bootstrap.cc:654] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.814992 1184 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:16.815156 1184 ts_tablet_manager.cc:1397] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250110 02:51:16.817286 525 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 452223af5f6a4836a0f582d394a9453e
I20250110 02:51:16.817761 525 catalog_manager.cc:6596] ReplaceTablet: tablet 452223af5f6a4836a0f582d394a9453e deleted and replaced by tablet fc1324eeac9446279304f442e2fad488
I20250110 02:51:16.818053 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 32dd1e726f1a41d0894737b3ef3caf8f from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:16.822635 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:16.824100 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452223af5f6a4836a0f582d394a9453e already in progress: creating tablet
W20250110 02:51:16.825762 853 leader_election.cc:343] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:16.825928 853 leader_election.cc:304] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:16.826339 1685 raft_consensus.cc:2743] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:16.827184 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.827436 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:16.828105 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452223af5f6a4836a0f582d394a9453e already in progress: creating tablet
W20250110 02:51:16.829957 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452223af5f6a4836a0f582d394a9453e already in progress: creating tablet
I20250110 02:51:16.831143 1188 tablet_bootstrap.cc:492] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:16.831360 1188 ts_tablet_manager.cc:1397] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.011s
I20250110 02:51:16.831924 1181 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.837438 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_8a8d43a5e45c47f083ed451f5b30bf6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:16.837702 1181 tablet_bootstrap.cc:654] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:16.838039 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a8d43a5e45c47f083ed451f5b30bf6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.839792 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.840224 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.847406 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_97ca49cb31634f579d9d84e0d4fc8057"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.847882 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97ca49cb31634f579d9d84e0d4fc8057 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.853178 1188 raft_consensus.cc:357] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.858755 1588 raft_consensus.cc:491] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.858927 1588 raft_consensus.cc:513] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.859712 1588 leader_election.cc:290] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:16.860661 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 235ffe7bf29a4cbcb4faef6f3b2931ec consensus state has no leader
I20250110 02:51:16.861204 519 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2] alter complete (version 1)
I20250110 02:51:16.862224 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66603a0ee8c4082807c5a6aabe7e339" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:16.862636 518 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb595013e809480a814cb9c7aed856e6 consensus state has no leader
I20250110 02:51:16.867524 1635 raft_consensus.cc:491] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.867673 1635 raft_consensus.cc:513] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.868271 1635 leader_election.cc:290] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:16.869302 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.869776 1179 tablet_replica.cc:331] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:16.870003 1179 raft_consensus.cc:2238] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.870205 1179 raft_consensus.cc:2267] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.870868 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.871080 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.871157 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:16.871336 774 raft_consensus.cc:2463] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:16.871711 996 leader_election.cc:304] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:16.872125 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00ebf8debb9d4394839d1c1cb97820a5 consensus state has no leader
I20250110 02:51:16.872612 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:16.873173 517 catalog_manager.cc:6596] ReplaceTablet: tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 deleted and replaced by tablet deaa3527aa074e6ba94960c672aee028
I20250110 02:51:16.873389 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_23df132302f5414db477144cbb57b923" } modify_external_catalogs: true
I20250110 02:51:16.873541 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_23df132302f5414db477144cbb57b923" } modify_external_catalogs: true
I20250110 02:51:16.873802 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bd6d4c4631994a5792200ba4d4c906f0 (table default.Table_742e66ce36c0463a978cb00de2f01135 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]). Expected version 3 got 0
I20250110 02:51:16.873837 1635 raft_consensus.cc:2798] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:16.874079 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:16.874287 518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0" } modify_external_catalogs: true
I20250110 02:51:16.874424 518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0" } modify_external_catalogs: true
I20250110 02:51:16.874446 1635 raft_consensus.cc:491] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:16.874557 1635 raft_consensus.cc:3054] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.874823 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c2f177ce9d6a420bbc77b33929ccaa7a" } modify_external_catalogs: true
I20250110 02:51:16.874941 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c2f177ce9d6a420bbc77b33929ccaa7a" } modify_external_catalogs: true
I20250110 02:51:16.875802 1179 ts_tablet_manager.cc:1905] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:16.876081 711 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.876092 526 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_72de93e5a855471e85416b410edf348d" } modify_external_catalogs: true
I20250110 02:51:16.876255 526 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_72de93e5a855471e85416b410edf348d" } modify_external_catalogs: true
I20250110 02:51:16.877187 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2e1a57f017f844aba03084fd604240cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.877663 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e1a57f017f844aba03084fd604240cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.878867 521 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b735b26f52fc4d4d9a3f706c984d008e" } modify_external_catalogs: true
I20250110 02:51:16.878909 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.879029 521 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_b735b26f52fc4d4d9a3f706c984d008e" } modify_external_catalogs: true
I20250110 02:51:16.879086 1635 raft_consensus.cc:513] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.874343 523 catalog_manager.cc:5526] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:16.879644 523 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]). Expected version 1 got 0
I20250110 02:51:16.880374 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:16.880631 774 raft_consensus.cc:3054] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.857544 1685 raft_consensus.cc:491] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:16.881858 1685 raft_consensus.cc:513] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.882463 1685 leader_election.cc:290] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:16.860834 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66603a0ee8c4082807c5a6aabe7e339" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.883087 709 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.883198 774 raft_consensus.cc:2463] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:16.883216 709 leader_election.cc:304] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:16.883260 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:16.883426 1061 raft_consensus.cc:2463] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:16.883594 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:16.883762 775 raft_consensus.cc:2463] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:16.883769 853 leader_election.cc:304] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:16.883963 1588 raft_consensus.cc:2743] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:16.884027 1685 raft_consensus.cc:2798] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:16.884115 1685 raft_consensus.cc:491] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:16.884182 1685 raft_consensus.cc:3054] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.870908 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:16.885228 994 leader_election.cc:343] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.886199 996 leader_election.cc:304] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:16.886528 1678 raft_consensus.cc:2798] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:16.886647 1678 raft_consensus.cc:695] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:16.886713 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:16.886703 1685 raft_consensus.cc:513] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.886906 1678 consensus_queue.cc:237] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.887557 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:16.887729 775 raft_consensus.cc:3054] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.888123 1179 ts_tablet_manager.cc:1918] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.888262 1179 log.cc:1198] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:16.888687 1179 ts_tablet_manager.cc:1939] T 60e534bf66f044b88d8cda0f3a1228f4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.889513 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:16.889686 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 60e534bf66f044b88d8cda0f3a1228f4 (table default.Table_8df25635dd7e4d58818dc14fc678e221 [id=4893b36e085445c4a454cf47ebe1955b]) successfully deleted
W20250110 02:51:16.890061 994 leader_election.cc:343] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:16.890190 775 raft_consensus.cc:2463] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:16.890363 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 already in progress: creating tablet
I20250110 02:51:16.890522 1635 leader_election.cc:290] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:16.890801 853 leader_election.cc:304] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:16.891134 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.891217 1685 leader_election.cc:290] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:16.891357 1685 raft_consensus.cc:2798] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:16.891464 1685 raft_consensus.cc:695] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:16.891709 1685 consensus_queue.cc:237] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.893041 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.894941 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09235548b5d4e10a01cbe05d1c94841" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:16.895114 1061 raft_consensus.cc:3054] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:16.896248 1181 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.896449 1181 ts_tablet_manager.cc:1397] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20250110 02:51:16.897293 1181 raft_consensus.cc:357] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.897509 1181 raft_consensus.cc:383] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.897584 1181 raft_consensus.cc:738] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.897684 1061 raft_consensus.cc:2463] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:16.898021 1181 consensus_queue.cc:260] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.898190 1179 tablet_replica.cc:331] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:16.898495 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 already in progress: creating tablet
I20250110 02:51:16.899081 1181 ts_tablet_manager.cc:1428] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:16.899619 1188 raft_consensus.cc:383] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.900115 1188 raft_consensus.cc:738] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.898424 1179 raft_consensus.cc:2238] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.900708 1179 raft_consensus.cc:2267] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.899775 1181 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.900504 1188 consensus_queue.cc:260] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.903589 1181 tablet_bootstrap.cc:654] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.903594 1179 ts_tablet_manager.cc:1905] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.907917 1188 ts_tablet_manager.cc:1428] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20250110 02:51:16.908543 1188 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:16.911463 522 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_8d7fe23e0afe4dfdac17778b492f2c55 [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 3 got 0
I20250110 02:51:16.912644 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.913117 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.915678 1181 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.915815 1181 ts_tablet_manager.cc:1397] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250110 02:51:16.915979 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:16.916466 1181 raft_consensus.cc:357] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.916635 1181 raft_consensus.cc:383] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.916713 1181 raft_consensus.cc:738] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.916620 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_410dc5667b5446fca70ebd92193bbc32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:16.916927 1181 consensus_queue.cc:260] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:16.917168 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_410dc5667b5446fca70ebd92193bbc32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.917491 1181 ts_tablet_manager.cc:1428] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:16.917765 1181 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.918475 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.919744 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_f2fa05ad4d3d4931b4b42b8682420ce9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.920225 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.921180 1181 tablet_bootstrap.cc:654] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:16.923275 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:16.923949 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:16.924471 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:16.925336 519 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9b1e53691288443193e1896ddc095952" } new_table_name: "default.Table_0105e3f6dea94fa2a6c83ea8b63220b1" modify_external_catalogs: true
I20250110 02:51:16.927788 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 2abd85a070f74e9bb5dc4026935edcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.928122 1178 tablet_replica.cc:331] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:16.928313 1178 raft_consensus.cc:2238] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.928447 1178 raft_consensus.cc:2267] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.928871 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:16.929162 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602ade49b7442d6b8e6f4b85d2e80a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 302af84fd8374ffa96a7503b54849536) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:16.929960 1178 ts_tablet_manager.cc:1905] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.934940 520 catalog_manager.cc:5526] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:16.935449 520 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_8d7fe23e0afe4dfdac17778b492f2c55 [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 3 got 0
I20250110 02:51:16.935649 520 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]). Expected version 1 got 0
I20250110 02:51:16.935861 1178 ts_tablet_manager.cc:1918] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.936010 1178 log.cc:1198] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2abd85a070f74e9bb5dc4026935edcb5
I20250110 02:51:16.936344 1178 ts_tablet_manager.cc:1939] T 2abd85a070f74e9bb5dc4026935edcb5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:16.939230 1178 tablet_replica.cc:331] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:16.939400 1178 raft_consensus.cc:2238] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.939558 1178 raft_consensus.cc:2267] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.942369 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2abd85a070f74e9bb5dc4026935edcb5 (table default.Table_a788c9c301f14b34b873006c6715a243 [id=4d2cfdd4eb154375945d51f3e591f0eb]) successfully deleted
I20250110 02:51:16.949193 1181 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:16.949376 1181 ts_tablet_manager.cc:1397] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20250110 02:51:16.950248 1181 raft_consensus.cc:357] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.950524 1181 raft_consensus.cc:383] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.950603 1181 raft_consensus.cc:738] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.950862 1181 consensus_queue.cc:260] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:16.951515 1181 ts_tablet_manager.cc:1428] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:16.952971 1179 ts_tablet_manager.cc:1918] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:16.953130 1179 log.cc:1198] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:16.954473 1181 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:16.955186 1651 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:16.955281 1651 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: loading cluster ID for follower catalog manager: success
I20250110 02:51:16.956270 1181 tablet_bootstrap.cc:654] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.957666 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_61c4e10750aa4566916853b2c35ed9fb" } new_table_name: "default.Table_bfbd91df5def489aa8830a6d7345e498" modify_external_catalogs: true
I20250110 02:51:16.967207 1179 ts_tablet_manager.cc:1939] T f66603a0ee8c4082807c5a6aabe7e339 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:16.968294 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f66603a0ee8c4082807c5a6aabe7e339 (table default.Table_7ab0055319bd4962a0267ce290806e0f [id=8488c24625ca46ba8f0d2405ab565feb]) successfully deleted
I20250110 02:51:16.974694 1179 tablet_replica.cc:331] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:16.974838 1178 ts_tablet_manager.cc:1905] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.974889 1179 raft_consensus.cc:2238] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:16.975065 1179 raft_consensus.cc:2267] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:16.975777 1179 ts_tablet_manager.cc:1905] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:16.980240 1188 tablet_bootstrap.cc:654] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:16.982127 1184 raft_consensus.cc:357] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.991945 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_c0c89321da6c41cbbf765c7562f60e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:16.992518 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c89321da6c41cbbf765c7562f60e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:16.994271 1184 raft_consensus.cc:383] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:16.994407 1184 raft_consensus.cc:738] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:16.994660 1184 consensus_queue.cc:260] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:16.995194 1184 ts_tablet_manager.cc:1428] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.180s user 0.002s sys 0.000s
I20250110 02:51:16.998486 1179 ts_tablet_manager.cc:1918] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.000655 1179 log.cc:1198] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:17.001080 1179 ts_tablet_manager.cc:1939] T 5e183122d13a4c38988bf2fe6f7a3694 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:17.001879 519 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9abd7180bf64bb5991be5f67ea42fbf has no consensus state
I20250110 02:51:17.002177 519 master_service.cc:946] ReplaceTablet: received request to replace tablet 2abd85a070f74e9bb5dc4026935edcb5 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:17.007377 1651 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: acquiring CA information for follower catalog manager: success
I20250110 02:51:17.025635 519 master_service.cc:946] ReplaceTablet: received request to replace tablet e744cdc1fcb94d46b48b684763ecf1bf from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:17.026988 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 5e183122d13a4c38988bf2fe6f7a3694 (table default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]) successfully deleted
I20250110 02:51:17.027419 1179 tablet_replica.cc:331] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.027617 1179 raft_consensus.cc:2238] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.027758 1179 raft_consensus.cc:2267] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.028251 1179 ts_tablet_manager.cc:1905] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.031510 1184 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:17.032568 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 already in progress: creating tablet
I20250110 02:51:17.033103 1184 tablet_bootstrap.cc:654] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.035398 1179 ts_tablet_manager.cc:1918] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.035567 1179 log.cc:1198] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:17.035957 1179 ts_tablet_manager.cc:1939] T 9e72ed2d31a8467f8c092b48d593c1cc P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.038514 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 9e72ed2d31a8467f8c092b48d593c1cc (table default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]) successfully deleted
I20250110 02:51:17.040364 1188 tablet_bootstrap.cc:492] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:17.040498 1188 ts_tablet_manager.cc:1397] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.132s user 0.001s sys 0.002s
W20250110 02:51:17.041168 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:17.041224 1188 raft_consensus.cc:357] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.041407 1188 raft_consensus.cc:383] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.041509 1188 raft_consensus.cc:738] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.041734 1188 consensus_queue.cc:260] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.057480 1651 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 02:51:17.066454 523 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43] alter complete (version 3)
W20250110 02:51:17.066536 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24cf81039d5e4a47b426ea5d2f21acc2 consensus state has no leader
I20250110 02:51:17.067060 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_09cf33925be04c35b8a1304633c52217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.067689 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09cf33925be04c35b8a1304633c52217 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.067946 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_1163b9629c7143d8a9aa6252b1c8b658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.068189 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_d8da12d0416f480794b1d69b6c62f386" } new_table_name: "default.Table_6e6ef42caec74748b566727e29ff4336" modify_external_catalogs: true
W20250110 02:51:17.068398 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1163b9629c7143d8a9aa6252b1c8b658 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.071074 520 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9064b36ee7684e2682fc53f68bcf1f47" } new_table_name: "default.Table_e3e5bb662c644b5c8d10bd0f63ffe818" modify_external_catalogs: true
I20250110 02:51:17.072098 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.074031 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.075376 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.075587 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.075820 1179 tablet_replica.cc:331] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.076035 1179 raft_consensus.cc:2238] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.076200 1179 raft_consensus.cc:2267] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.076516 1180 tablet_replica.cc:331] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:17.076730 1180 raft_consensus.cc:2238] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.076911 1180 raft_consensus.cc:2267] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.077556 1180 ts_tablet_manager.cc:1905] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.077828 1188 ts_tablet_manager.cc:1428] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250110 02:51:17.079907 1188 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:17.084874 1188 tablet_bootstrap.cc:654] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.085620 1180 ts_tablet_manager.cc:1918] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.085755 1180 log.cc:1198] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/89f6797b1076468a93fb0b510c553bed
I20250110 02:51:17.086134 1180 ts_tablet_manager.cc:1939] T 89f6797b1076468a93fb0b510c553bed P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:17.089509 1179 ts_tablet_manager.cc:1905] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.093232 1184 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.093385 1184 ts_tablet_manager.cc:1397] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250110 02:51:17.094204 1184 raft_consensus.cc:357] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.094440 1184 raft_consensus.cc:383] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.094525 1184 raft_consensus.cc:738] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.094739 1184 consensus_queue.cc:260] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.095245 1184 ts_tablet_manager.cc:1428] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:17.095504 1184 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.096853 1178 ts_tablet_manager.cc:1918] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.096987 1178 log.cc:1198] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1602ade49b7442d6b8e6f4b85d2e80a1
I20250110 02:51:17.097424 1178 ts_tablet_manager.cc:1939] T 1602ade49b7442d6b8e6f4b85d2e80a1 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:17.098232 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cb20d47ace4fd49f9dc4e526cab702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.100803 1179 ts_tablet_manager.cc:1918] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.100963 1179 log.cc:1198] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/0127de45264f4660b843734a530eb310
I20250110 02:51:17.101362 1179 ts_tablet_manager.cc:1939] T 0127de45264f4660b843734a530eb310 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.102469 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 0127de45264f4660b843734a530eb310 (table default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:17.103436 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 89f6797b1076468a93fb0b510c553bed (table default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]) successfully deleted
I20250110 02:51:17.103729 1184 tablet_bootstrap.cc:654] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.103844 1179 tablet_replica.cc:331] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.104202 1180 tablet_replica.cc:331] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:17.104478 1179 raft_consensus.cc:2238] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.104652 1179 raft_consensus.cc:2267] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.105470 1179 ts_tablet_manager.cc:1905] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.105631 1180 raft_consensus.cc:2238] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.105835 1180 raft_consensus.cc:2267] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.108445 525 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 5 got 0
I20250110 02:51:17.111609 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_c4155bb1db484441b38ef15357e8f40f" } new_table_name: "default.Table_4ffa32dac1064dc68c031399c84f8f8c" modify_external_catalogs: true
I20250110 02:51:17.112440 1179 ts_tablet_manager.cc:1918] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.112577 1179 log.cc:1198] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:17.112920 1179 ts_tablet_manager.cc:1939] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.113915 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (table default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]) successfully deleted
I20250110 02:51:17.114214 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.117493 1181 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:17.117643 1181 ts_tablet_manager.cc:1397] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.163s user 0.026s sys 0.056s
I20250110 02:51:17.118470 1181 raft_consensus.cc:357] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.118760 1181 raft_consensus.cc:383] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.118842 1181 raft_consensus.cc:738] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.125679 1180 ts_tablet_manager.cc:1905] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.130977 1179 tablet_replica.cc:331] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.131106 1181 consensus_queue.cc:260] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.131258 1179 raft_consensus.cc:2238] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.131529 1179 raft_consensus.cc:2267] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.132280 1179 ts_tablet_manager.cc:1905] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.132741 1184 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.132886 1184 ts_tablet_manager.cc:1397] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250110 02:51:17.133653 1184 raft_consensus.cc:357] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.133837 1184 raft_consensus.cc:383] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.133916 1184 raft_consensus.cc:738] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.134114 1184 consensus_queue.cc:260] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.134634 1184 ts_tablet_manager.cc:1428] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:17.134933 1184 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.136516 1184 tablet_bootstrap.cc:654] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.139001 1588 raft_consensus.cc:491] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.139043 1181 ts_tablet_manager.cc:1428] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.021s user 0.000s sys 0.003s
I20250110 02:51:17.139148 1588 raft_consensus.cc:513] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.139717 1588 leader_election.cc:290] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.140290 1181 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:17.140990 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:17.142012 709 leader_election.cc:343] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.144330 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:17.144955 711 leader_election.cc:343] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.145097 711 leader_election.cc:304] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.145375 1588 raft_consensus.cc:2743] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.147403 1179 ts_tablet_manager.cc:1918] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.147529 1179 log.cc:1198] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:17.148002 1179 ts_tablet_manager.cc:1939] T d10ace37b96844328ccdb821dad3c5ea P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.148254 1181 tablet_bootstrap.cc:654] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.148006 1184 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.148653 1184 ts_tablet_manager.cc:1397] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250110 02:51:17.149152 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d10ace37b96844328ccdb821dad3c5ea (table default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]) successfully deleted
I20250110 02:51:17.149363 1184 raft_consensus.cc:357] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.149542 1184 raft_consensus.cc:383] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.149616 1184 raft_consensus.cc:738] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.149858 1184 consensus_queue.cc:260] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.150409 1184 ts_tablet_manager.cc:1428] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:17.150696 1184 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.152766 1184 tablet_bootstrap.cc:654] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.157833 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_eb40720cee0c4ef08e2c5a49f1374c8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.158900 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb40720cee0c4ef08e2c5a49f1374c8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:17.164906 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 820975aa259f4274ba6c19398425e94b consensus state has no leader
I20250110 02:51:17.166679 1678 raft_consensus.cc:491] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.166939 1678 raft_consensus.cc:513] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.167734 1678 leader_election.cc:290] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:17.168202 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:17.168402 775 raft_consensus.cc:2463] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:17.168673 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.168774 996 leader_election.cc:304] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:17.169027 1678 raft_consensus.cc:2798] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:17.169106 1678 raft_consensus.cc:491] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:17.169184 1678 raft_consensus.cc:3054] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:17.169521 994 leader_election.cc:343] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.171698 1678 raft_consensus.cc:513] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.173077 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:17.173251 775 raft_consensus.cc:3054] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.174106 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:17.174665 994 leader_election.cc:343] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.176115 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e4c35b6a442b4b95a1341303de60226f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.176579 775 raft_consensus.cc:2463] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:17.176853 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4c35b6a442b4b95a1341303de60226f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.176957 996 leader_election.cc:304] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:17.177253 1635 raft_consensus.cc:2798] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:17.177397 1635 raft_consensus.cc:695] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:17.177646 1635 consensus_queue.cc:237] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.180214 1678 leader_election.cc:290] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:17.195914 520 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 467f87955c4a4d27a6f0483aa109b922 consensus state has no leader
I20250110 02:51:17.202473 520 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_161f237741c449f1b10e9b240cca7ef2" } modify_external_catalogs: true
I20250110 02:51:17.202720 520 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_161f237741c449f1b10e9b240cca7ef2" } modify_external_catalogs: true
I20250110 02:51:17.221479 519 catalog_manager.cc:5526] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:17.222025 519 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bd6d4c4631994a5792200ba4d4c906f0 (table default.Table_742e66ce36c0463a978cb00de2f01135 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]). Expected version 3 got 0
I20250110 02:51:17.222365 519 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:17.224498 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b4c2a19225ed477281b7ce60ae25b084 (DEFAULT_TABLE table=default.Table_3f51e7f8aaa7470fbeeb5459695efab1 [id=4b8aabec28ba49388402a7d16de036c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.224920 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c2a19225ed477281b7ce60ae25b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.226642 1037 tablet_service.cc:1467] Processing CreateTablet for tablet b5134444da58427aa97e166e44a3a648 (DEFAULT_TABLE table=default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 [id=c758d6f1cb1e4e48bee42c36a5df0e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.227016 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5134444da58427aa97e166e44a3a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.230659 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 07acc4681ae94a069003ed75fcce5141 (DEFAULT_TABLE table=default.Table_5156f6b732b14eb6807a520bdfa15b62 [id=995906957594490ba9474338a3e342d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.230969 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07acc4681ae94a069003ed75fcce5141. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.234731 756 tablet_service.cc:1467] Processing CreateTablet for tablet b4c2a19225ed477281b7ce60ae25b084 (DEFAULT_TABLE table=default.Table_3f51e7f8aaa7470fbeeb5459695efab1 [id=4b8aabec28ba49388402a7d16de036c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.235090 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c2a19225ed477281b7ce60ae25b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.237866 744 tablet_service.cc:1467] Processing CreateTablet for tablet b5134444da58427aa97e166e44a3a648 (DEFAULT_TABLE table=default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 [id=c758d6f1cb1e4e48bee42c36a5df0e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.238215 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5134444da58427aa97e166e44a3a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.244920 889 tablet_service.cc:1467] Processing CreateTablet for tablet b4c2a19225ed477281b7ce60ae25b084 (DEFAULT_TABLE table=default.Table_3f51e7f8aaa7470fbeeb5459695efab1 [id=4b8aabec28ba49388402a7d16de036c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.245352 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c2a19225ed477281b7ce60ae25b084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.248768 898 tablet_service.cc:1467] Processing CreateTablet for tablet b5134444da58427aa97e166e44a3a648 (DEFAULT_TABLE table=default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 [id=c758d6f1cb1e4e48bee42c36a5df0e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.249118 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5134444da58427aa97e166e44a3a648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.250795 887 tablet_service.cc:1467] Processing CreateTablet for tablet 07acc4681ae94a069003ed75fcce5141 (DEFAULT_TABLE table=default.Table_5156f6b732b14eb6807a520bdfa15b62 [id=995906957594490ba9474338a3e342d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.251101 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07acc4681ae94a069003ed75fcce5141. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.253412 1178 tablet_replica.cc:331] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:17.253744 1178 raft_consensus.cc:2238] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.254631 882 tablet_service.cc:1467] Processing CreateTablet for tablet a254ed6c239249c6982429eb5e142031 (DEFAULT_TABLE table=default.Table_7cebf0b5d59d4f068fc8dce07001bd12 [id=f686f55e175d4a0bbdaa0d8a98d63791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.255160 893 tablet_service.cc:1467] Processing CreateTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.255241 1188 tablet_bootstrap.cc:492] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:17.255409 1188 ts_tablet_manager.cc:1397] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.176s user 0.024s sys 0.035s
I20250110 02:51:17.255633 884 tablet_service.cc:1467] Processing CreateTablet for tablet 6694671f422c4be7b545304a38b17edd (DEFAULT_TABLE table=default.Table_68a6c97dd17c47b28f987d6e8e778e37 [id=4d8aac2aeda44055ab88ed9f82e8efeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.256062 883 tablet_service.cc:1467] Processing CreateTablet for tablet 78d9140c04a14afc96d43f3bc93488e9 (DEFAULT_TABLE table=default.Table_5131fbd504e04f50b92a6a1fb6d0c055 [id=7c940859b1954178bfcb14fb882668f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.256258 1188 raft_consensus.cc:357] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.256470 1188 raft_consensus.cc:383] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.256539 1188 raft_consensus.cc:738] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.256633 886 tablet_service.cc:1467] Processing CreateTablet for tablet 78c0808330e848249e1d7eb71ab0d362 (DEFAULT_TABLE table=default.Table_375ea34c1b6d471ca632286504f259f0 [id=5d041eb06ad54278b7a327c69bbae96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.256783 1188 consensus_queue.cc:260] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.257073 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc408e62034459815486648565e7ce (DEFAULT_TABLE table=default.Table_37fa37bb314c4af39a4bb95697714acb [id=f54f0ab6a3d140ae9ceac1818d68f7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.257484 896 tablet_service.cc:1467] Processing CreateTablet for tablet 67294cdecffc4ad4b35e000a3e01c9be (DEFAULT_TABLE table=default.Table_c5d78661dad44d9f85ba6c91486185de [id=a6a89e8f33c74e61b95de2207ddbd585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.257858 880 tablet_service.cc:1467] Processing CreateTablet for tablet 656aa84a130c42768e969b9322fbd8a5 (DEFAULT_TABLE table=default.Table_68d7c1c889464468adcd9153caef7ecc [id=497a56d44a114093934dbb9fde1b9f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.258179 891 tablet_service.cc:1467] Processing CreateTablet for tablet c975136880034808b58276e06fbd94f9 (DEFAULT_TABLE table=default.Table_3b60f335e3574a5bbf7ab20a93a64bed [id=b00159b02a3a4c2d877a7e8c7bf1fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.258510 1188 ts_tablet_manager.cc:1428] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:17.259927 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9497c76dcffa45268f705ca6423db41c (DEFAULT_TABLE table=default.Table_8835f1dac47741f69e9a48aed136cde0 [id=e8a9e62e18bf419e807e689558db87ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.253958 1178 raft_consensus.cc:2267] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.262904 1178 ts_tablet_manager.cc:1905] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:17.263422 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 81337.1us
I20250110 02:51:17.266474 754 tablet_service.cc:1467] Processing CreateTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.266656 740 tablet_service.cc:1467] Processing CreateTablet for tablet 07acc4681ae94a069003ed75fcce5141 (DEFAULT_TABLE table=default.Table_5156f6b732b14eb6807a520bdfa15b62 [id=995906957594490ba9474338a3e342d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.266888 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82bd4307e5fe4f61b9308005cf3446e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.267194 753 tablet_service.cc:1467] Processing CreateTablet for tablet a254ed6c239249c6982429eb5e142031 (DEFAULT_TABLE table=default.Table_7cebf0b5d59d4f068fc8dce07001bd12 [id=f686f55e175d4a0bbdaa0d8a98d63791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.267541 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a254ed6c239249c6982429eb5e142031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.271459 741 tablet_service.cc:1467] Processing CreateTablet for tablet 6694671f422c4be7b545304a38b17edd (DEFAULT_TABLE table=default.Table_68a6c97dd17c47b28f987d6e8e778e37 [id=4d8aac2aeda44055ab88ed9f82e8efeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.271816 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6694671f422c4be7b545304a38b17edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.271654 522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.272289 522 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.275032 743 tablet_service.cc:1467] Processing CreateTablet for tablet 78d9140c04a14afc96d43f3bc93488e9 (DEFAULT_TABLE table=default.Table_5131fbd504e04f50b92a6a1fb6d0c055 [id=7c940859b1954178bfcb14fb882668f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.275365 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78d9140c04a14afc96d43f3bc93488e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:17.277884 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e03d53860ab744968693cf96467c3a19 consensus state has no leader
I20250110 02:51:17.278147 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9497c76dcffa45268f705ca6423db41c (DEFAULT_TABLE table=default.Table_8835f1dac47741f69e9a48aed136cde0 [id=e8a9e62e18bf419e807e689558db87ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.278463 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9497c76dcffa45268f705ca6423db41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.282406 525 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:17.283653 525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_633f75bba62141e9a0075b23b2ec4bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.283841 1188 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:17.284600 525 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_633f75bba62141e9a0075b23b2ec4bc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.285545 1188 tablet_bootstrap.cc:654] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.285866 1680 raft_consensus.cc:491] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.286013 1029 tablet_service.cc:1467] Processing CreateTablet for tablet aa2b5a305db34fedbbbf0d666e6c28cb (DEFAULT_TABLE table=default.Table_a993dbbdb6714184867df68d17523ca7 [id=8cf03d1b06f24347a2f8c4132f135d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.286761 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2b5a305db34fedbbbf0d666e6c28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.286208 1022 tablet_service.cc:1467] Processing CreateTablet for tablet a254ed6c239249c6982429eb5e142031 (DEFAULT_TABLE table=default.Table_7cebf0b5d59d4f068fc8dce07001bd12 [id=f686f55e175d4a0bbdaa0d8a98d63791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.287056 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a254ed6c239249c6982429eb5e142031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.287312 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f4751d0f64e39b975529c6ac28d8a (DEFAULT_TABLE table=default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf [id=cf302e964fd54aa7bb808f9801488d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.287699 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2f4751d0f64e39b975529c6ac28d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.288218 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.288558 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82bd4307e5fe4f61b9308005cf3446e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.290640 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 6694671f422c4be7b545304a38b17edd (DEFAULT_TABLE table=default.Table_68a6c97dd17c47b28f987d6e8e778e37 [id=4d8aac2aeda44055ab88ed9f82e8efeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.290992 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6694671f422c4be7b545304a38b17edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.287061 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e9abd7180bf64bb5991be5f67ea42fbf (DEFAULT_TABLE table=default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.292971 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9abd7180bf64bb5991be5f67ea42fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.294644 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 9497c76dcffa45268f705ca6423db41c (DEFAULT_TABLE table=default.Table_8835f1dac47741f69e9a48aed136cde0 [id=e8a9e62e18bf419e807e689558db87ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.294962 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9497c76dcffa45268f705ca6423db41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.286767 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 78d9140c04a14afc96d43f3bc93488e9 (DEFAULT_TABLE table=default.Table_5131fbd504e04f50b92a6a1fb6d0c055 [id=7c940859b1954178bfcb14fb882668f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.298302 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07acc4681ae94a069003ed75fcce5141. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.304869 524 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:17.305765 524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_5e1c1dd369d94a688dabf41fda3795b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.306015 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:17.306272 524 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1c1dd369d94a688dabf41fda3795b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.306253 897 tablet_service.cc:1467] Processing CreateTablet for tablet e9abd7180bf64bb5991be5f67ea42fbf (DEFAULT_TABLE table=default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.306591 895 tablet_service.cc:1467] Processing CreateTablet for tablet aa2b5a305db34fedbbbf0d666e6c28cb (DEFAULT_TABLE table=default.Table_a993dbbdb6714184867df68d17523ca7 [id=8cf03d1b06f24347a2f8c4132f135d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.306768 885 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0f754966c04074a568c05063591990 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.306854 523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_2009672da4c945ec8fbbac6f5fe0988a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.306984 894 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f4751d0f64e39b975529c6ac28d8a (DEFAULT_TABLE table=default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf [id=cf302e964fd54aa7bb808f9801488d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.307255 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1602ade49b7442d6b8e6f4b85d2e80a1 (table default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]) successfully deleted
W20250110 02:51:17.307353 523 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2009672da4c945ec8fbbac6f5fe0988a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.307880 526 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:17.308071 754 tablet_service.cc:1467] Processing CreateTablet for tablet 78c0808330e848249e1d7eb71ab0d362 (DEFAULT_TABLE table=default.Table_375ea34c1b6d471ca632286504f259f0 [id=5d041eb06ad54278b7a327c69bbae96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.308461 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c0808330e848249e1d7eb71ab0d362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.308712 526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_7fdb03687ec14267bfee2826ddb5a13b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.309257 526 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fdb03687ec14267bfee2826ddb5a13b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.309448 745 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc408e62034459815486648565e7ce (DEFAULT_TABLE table=default.Table_37fa37bb314c4af39a4bb95697714acb [id=f54f0ab6a3d140ae9ceac1818d68f7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.310748 1188 tablet_bootstrap.cc:492] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:17.310876 1188 ts_tablet_manager.cc:1397] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20250110 02:51:17.311115 748 tablet_service.cc:1467] Processing CreateTablet for tablet 656aa84a130c42768e969b9322fbd8a5 (DEFAULT_TABLE table=default.Table_68d7c1c889464468adcd9153caef7ecc [id=497a56d44a114093934dbb9fde1b9f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.311647 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.311919 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc408e62034459815486648565e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.313324 1188 raft_consensus.cc:357] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.314034 1188 raft_consensus.cc:383] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.314296 1188 raft_consensus.cc:738] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.315027 1188 consensus_queue.cc:260] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.317016 1188 ts_tablet_manager.cc:1428] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:17.317034 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.321616 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f4751d0f64e39b975529c6ac28d8a (DEFAULT_TABLE table=default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf [id=cf302e964fd54aa7bb808f9801488d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.321909 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a254ed6c239249c6982429eb5e142031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.322124 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0f754966c04074a568c05063591990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.322315 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.322549 1716 raft_consensus.cc:491] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.322767 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c975136880034808b58276e06fbd94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.322670 1716 raft_consensus.cc:513] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.323163 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c0808330e848249e1d7eb71ab0d362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.323346 1716 leader_election.cc:290] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.323563 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78d9140c04a14afc96d43f3bc93488e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.323671 521 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:17.323921 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2f4751d0f64e39b975529c6ac28d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.324259 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc408e62034459815486648565e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.324517 747 tablet_service.cc:1467] Processing CreateTablet for tablet c975136880034808b58276e06fbd94f9 (DEFAULT_TABLE table=default.Table_3b60f335e3574a5bbf7ab20a93a64bed [id=b00159b02a3a4c2d877a7e8c7bf1fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.324815 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67294cdecffc4ad4b35e000a3e01c9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.324925 521 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:17.325038 755 tablet_service.cc:1467] Processing CreateTablet for tablet aa2b5a305db34fedbbbf0d666e6c28cb (DEFAULT_TABLE table=default.Table_a993dbbdb6714184867df68d17523ca7 [id=8cf03d1b06f24347a2f8c4132f135d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.325196 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6694671f422c4be7b545304a38b17edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.325434 746 tablet_service.cc:1467] Processing CreateTablet for tablet 67294cdecffc4ad4b35e000a3e01c9be (DEFAULT_TABLE table=default.Table_c5d78661dad44d9f85ba6c91486185de [id=a6a89e8f33c74e61b95de2207ddbd585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.325547 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9497c76dcffa45268f705ca6423db41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.325562 521 master_service.cc:946] ReplaceTablet: received request to replace tablet 940445840db3441598eba57d3ccb8c9c from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:17.325866 751 tablet_service.cc:1467] Processing CreateTablet for tablet e9abd7180bf64bb5991be5f67ea42fbf (DEFAULT_TABLE table=default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.326016 521 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_420beb896535455a87ee0a8c6a6049d4" } new_table_name: "default.Table_e49b14cd5bac4073be1ad6ffe44eb92c" modify_external_catalogs: true
I20250110 02:51:17.326238 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82bd4307e5fe4f61b9308005cf3446e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.326671 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa84a130c42768e969b9322fbd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.327121 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2b5a305db34fedbbbf0d666e6c28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.327527 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9abd7180bf64bb5991be5f67ea42fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.328897 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.334427 750 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0f754966c04074a568c05063591990 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.336921 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (DEFAULT_TABLE table=default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.337018 1680 raft_consensus.cc:513] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.337422 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.337808 1680 leader_election.cc:290] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:17.339208 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa76357abae4a95b8fb3af47288e43a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:17.339865 994 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.336956 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0f754966c04074a568c05063591990 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.342878 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0f754966c04074a568c05063591990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.341423 1188 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:17.348488 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78d9140c04a14afc96d43f3bc93488e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.348609 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 78c0808330e848249e1d7eb71ab0d362 (DEFAULT_TABLE table=default.Table_375ea34c1b6d471ca632286504f259f0 [id=5d041eb06ad54278b7a327c69bbae96a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.348659 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa84a130c42768e969b9322fbd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.348862 1038 tablet_service.cc:1467] Processing CreateTablet for tablet 67294cdecffc4ad4b35e000a3e01c9be (DEFAULT_TABLE table=default.Table_c5d78661dad44d9f85ba6c91486185de [id=a6a89e8f33c74e61b95de2207ddbd585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.348913 1678 raft_consensus.cc:491] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.349191 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2f4751d0f64e39b975529c6ac28d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.350359 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c975136880034808b58276e06fbd94f9 (DEFAULT_TABLE table=default.Table_3b60f335e3574a5bbf7ab20a93a64bed [id=b00159b02a3a4c2d877a7e8c7bf1fdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.350440 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.350498 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 656aa84a130c42768e969b9322fbd8a5 (DEFAULT_TABLE table=default.Table_68d7c1c889464468adcd9153caef7ecc [id=497a56d44a114093934dbb9fde1b9f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.351176 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c0808330e848249e1d7eb71ab0d362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.353649 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.357450 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67294cdecffc4ad4b35e000a3e01c9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.358364 1588 raft_consensus.cc:491] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.359306 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.362267 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0f754966c04074a568c05063591990. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:17.362995 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 09f01daf4c67462ba75e1d0cf56cef12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f01daf4c67462ba75e1d0cf56cef12 already in progress: creating tablet
W20250110 02:51:17.363317 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
I20250110 02:51:17.363301 1588 raft_consensus.cc:513] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.363878 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:17.367525 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.367769 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9abd7180bf64bb5991be5f67ea42fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.368614 1678 raft_consensus.cc:513] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.370935 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2b5a305db34fedbbbf0d666e6c28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.373229 1040 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.373620 1716 raft_consensus.cc:491] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.374274 1716 raft_consensus.cc:513] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.376526 518 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 1fb5412b78104d8085878c58b191ab27
I20250110 02:51:17.376499 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.377554 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:17.377830 775 raft_consensus.cc:2388] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
W20250110 02:51:17.378113 709 leader_election.cc:343] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.379176 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.379640 1179 tablet_replica.cc:331] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.379858 1179 raft_consensus.cc:2238] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.380020 1179 raft_consensus.cc:2267] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.380575 1179 ts_tablet_manager.cc:1905] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.381260 1032 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc408e62034459815486648565e7ce (DEFAULT_TABLE table=default.Table_37fa37bb314c4af39a4bb95697714acb [id=f54f0ab6a3d140ae9ceac1818d68f7bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.383401 1588 leader_election.cc:290] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.384526 1180 ts_tablet_manager.cc:1918] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.384934 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c975136880034808b58276e06fbd94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.387813 1716 leader_election.cc:290] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.388929 1680 raft_consensus.cc:491] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.389073 1680 raft_consensus.cc:513] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.391791 1680 leader_election.cc:290] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:17.391908 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f229f546a8d4a09b93f8b46c893d14a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.392758 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc408e62034459815486648565e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.393090 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.393260 518 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8d7fe23e0afe4dfdac17778b492f2c55" } new_table_name: "default.Table_26b5c54688724d26b6bf4f3587158e9c" modify_external_catalogs: true
I20250110 02:51:17.394409 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c975136880034808b58276e06fbd94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.395311 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa84a130c42768e969b9322fbd8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.395390 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361c332d66984cf7ab84d74e913a5bde" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.396198 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.396380 853 leader_election.cc:304] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:17.398119 1678 leader_election.cc:290] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:17.399907 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67294cdecffc4ad4b35e000a3e01c9be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:17.399961 994 leader_election.cc:343] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.400537 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.407655 1716 raft_consensus.cc:3054] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.409494 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.411648 517 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet 240b64d551a548e884ee2a7238c0363a
I20250110 02:51:17.412887 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.415887 1180 log.cc:1198] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:17.417342 897 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.418408 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:17.418167 711 leader_election.cc:343] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.419862 711 leader_election.cc:304] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.420697 1588 raft_consensus.cc:2743] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.422049 517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_18eb7bac666944ddaf84da77edd4462b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.422581 517 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18eb7bac666944ddaf84da77edd4462b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.422891 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.423775 853 leader_election.cc:304] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:17.437831 519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_0de0a6680c0f4531a75432a618576660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.438342 519 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0de0a6680c0f4531a75432a618576660 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.443153 520 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Sending DeleteTablet for 3 replicas of tablet dba0cb7970f1453dac256b255ed7cac0
I20250110 02:51:17.447212 756 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55720
W20250110 02:51:17.448241 513 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bb595013e809480a814cb9c7aed856e6 consensus state has no leader
I20250110 02:51:17.454399 520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_a171d415e02a4113afd448d388d8f124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.454926 520 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a171d415e02a4113afd448d388d8f124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.458799 1726 raft_consensus.cc:2743] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.461370 1588 raft_consensus.cc:491] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.462284 1588 raft_consensus.cc:513] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.462911 1588 leader_election.cc:290] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.464848 897 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.464892 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.465309 916 raft_consensus.cc:3054] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:17.465637 709 leader_election.cc:343] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.473529 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e10e7654cec495bb532fb00e73c020d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c7fbdab52014dddbf63cfef69b09b7b) from {username='slave'} at 127.0.0.1:33070
I20250110 02:51:17.479652 1179 ts_tablet_manager.cc:1918] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.482311 1179 log.cc:1198] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1fb5412b78104d8085878c58b191ab27
I20250110 02:51:17.482892 1179 ts_tablet_manager.cc:1939] T 1fb5412b78104d8085878c58b191ab27 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.483322 522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_3832746ca2454627a4b2495825f5223c" } new_table_name: "default.Table_8f3acd38298944faabb07e4e3b27e9ec" modify_external_catalogs: true
I20250110 02:51:17.483800 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1fb5412b78104d8085878c58b191ab27 (table default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]) successfully deleted
W20250110 02:51:17.484342 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 240b64d551a548e884ee2a7238c0363a already in progress: creating tablet
I20250110 02:51:17.488492 775 raft_consensus.cc:1270] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:17.489230 1726 consensus_queue.cc:1035] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:17.489658 512 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
I20250110 02:51:17.495268 525 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_350d7434def5474991dcc142163cf175" } new_table_name: "default.Table_f45dec555ce747f9be01f55fa8d8383f" modify_external_catalogs: true
I20250110 02:51:17.496732 1184 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.496869 1184 ts_tablet_manager.cc:1397] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.346s user 0.012s sys 0.053s
I20250110 02:51:17.497624 1184 raft_consensus.cc:357] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.497835 1184 raft_consensus.cc:383] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.498122 1184 raft_consensus.cc:738] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.498348 1184 consensus_queue.cc:260] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.499013 1184 ts_tablet_manager.cc:1428] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:17.499267 1184 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.500753 1184 tablet_bootstrap.cc:654] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.503252 1184 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.503386 1184 ts_tablet_manager.cc:1397] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:17.504072 1184 raft_consensus.cc:357] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.504307 1184 raft_consensus.cc:383] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.504384 1184 raft_consensus.cc:738] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:17.504635 994 leader_election.cc:343] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.504597 1184 consensus_queue.cc:260] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.505120 1184 ts_tablet_manager.cc:1428] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:17.505404 1184 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.507347 1184 tablet_bootstrap.cc:654] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.514534 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:17.516815 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 51273.5us
I20250110 02:51:17.518146 1180 ts_tablet_manager.cc:1939] T 019c0d60da5548bb86c93ccd89bf94ff P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:17.519459 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:55354
I20250110 02:51:17.521360 1061 raft_consensus.cc:1270] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:17.521673 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.523875 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 019c0d60da5548bb86c93ccd89bf94ff (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
W20250110 02:51:17.524186 711 leader_election.cc:343] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.524312 711 leader_election.cc:304] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.524560 1588 raft_consensus.cc:2743] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.524518 1726 consensus_queue.cc:1035] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250110 02:51:17.527269 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 09f01daf4c67462ba75e1d0cf56cef12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f01daf4c67462ba75e1d0cf56cef12 already in progress: creating tablet
I20250110 02:51:17.528517 1184 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.528631 1184 ts_tablet_manager.cc:1397] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250110 02:51:17.529266 1184 raft_consensus.cc:357] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.529553 1184 raft_consensus.cc:383] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.529623 1184 raft_consensus.cc:738] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:17.529636 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
I20250110 02:51:17.529930 1184 consensus_queue.cc:260] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.531037 1184 ts_tablet_manager.cc:1428] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:17.531286 1184 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.532719 1184 tablet_bootstrap.cc:654] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.531870 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:55354
W20250110 02:51:17.535859 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1fb5412b78104d8085878c58b191ab27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb5412b78104d8085878c58b191ab27 already in progress: creating tablet
W20250110 02:51:17.536088 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 240b64d551a548e884ee2a7238c0363a already in progress: creating tablet
I20250110 02:51:17.536553 1180 tablet_replica.cc:331] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:17.536751 1180 raft_consensus.cc:2238] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.541193 1716 raft_consensus.cc:2743] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.544085 917 raft_consensus.cc:1270] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Refusing update from remote peer 8899dd1aeb384e4ea4523f5fd225c888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:17.546690 1734 consensus_queue.cc:1035] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250110 02:51:17.555428 1180 raft_consensus.cc:2267] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.560406 1178 ts_tablet_manager.cc:1918] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.561793 1184 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.561950 1184 ts_tablet_manager.cc:1397] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20250110 02:51:17.562781 1184 raft_consensus.cc:357] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.562960 1184 raft_consensus.cc:383] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.563014 1184 raft_consensus.cc:738] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.563838 1184 consensus_queue.cc:260] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.564450 1184 ts_tablet_manager.cc:1428] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250110 02:51:17.564719 1184 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.564823 1588 raft_consensus.cc:491] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.564978 1588 raft_consensus.cc:513] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.565698 1588 leader_election.cc:290] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.566336 1184 tablet_bootstrap.cc:654] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.567272 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:17.567901 711 leader_election.cc:343] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.566331 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.568089 1180 ts_tablet_manager.cc:1905] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.567791 1188 tablet_bootstrap.cc:654] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.569321 1178 log.cc:1198] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/48cb20d47ace4fd49f9dc4e526cab702
W20250110 02:51:17.569631 709 leader_election.cc:343] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.569836 1178 ts_tablet_manager.cc:1939] T 48cb20d47ace4fd49f9dc4e526cab702 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:17.569808 709 leader_election.cc:304] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.570238 1588 raft_consensus.cc:2743] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.570401 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 48cb20d47ace4fd49f9dc4e526cab702 (table default.Table_163c6a20e5ad4f8398dc3a947c92bf9f [id=d854ebf27ed94ded9ee83b7b66010dbf]) successfully deleted
W20250110 02:51:17.574060 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 09f01daf4c67462ba75e1d0cf56cef12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09f01daf4c67462ba75e1d0cf56cef12 already in progress: creating tablet
W20250110 02:51:17.574868 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
W20250110 02:51:17.575703 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1fb5412b78104d8085878c58b191ab27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fb5412b78104d8085878c58b191ab27 already in progress: creating tablet
W20250110 02:51:17.576932 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 240b64d551a548e884ee2a7238c0363a already in progress: creating tablet
W20250110 02:51:17.578197 510 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
I20250110 02:51:17.579560 1178 tablet_replica.cc:331] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:17.579761 1178 raft_consensus.cc:2238] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.579957 1178 raft_consensus.cc:2267] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.585638 1178 ts_tablet_manager.cc:1905] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.588766 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:17.588980 774 raft_consensus.cc:2463] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:17.591099 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:17.591284 774 raft_consensus.cc:2463] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:17.594627 523 master_service.cc:946] ReplaceTablet: received request to replace tablet b375a932156d4abb908dbf327689b0f9 from {username='slave'} at 127.0.0.1:46680
W20250110 02:51:17.596418 996 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:17.600903 523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_8f8cd5df24f147c5ae832efed9aad406" } new_table_name: "default.Table_2b13b779fcfb4d5892ff5975b2cd98f0" modify_external_catalogs: true
I20250110 02:51:17.603986 996 leader_election.cc:304] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:17.604308 996 raft_consensus.cc:2681] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 02:51:17.604413 1184 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.604573 1184 ts_tablet_manager.cc:1397] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250110 02:51:17.605578 1180 ts_tablet_manager.cc:1918] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.606210 996 leader_election.cc:304] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:17.606508 1734 raft_consensus.cc:2798] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:17.606823 1184 raft_consensus.cc:357] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.607169 1734 raft_consensus.cc:491] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:17.607299 1734 raft_consensus.cc:3054] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.607618 1184 raft_consensus.cc:383] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.607734 1184 raft_consensus.cc:738] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.607839 1180 log.cc:1198] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:17.607972 1184 consensus_queue.cc:260] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.608497 1184 ts_tablet_manager.cc:1428] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:17.608755 1184 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:17.610275 521 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c0d22fd75fb474ebf07b624499a6f7b consensus state has no leader
I20250110 02:51:17.611513 1734 raft_consensus.cc:513] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.612452 1734 leader_election.cc:290] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:17.612725 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:17.613256 994 leader_election.cc:343] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.615226 1184 tablet_bootstrap.cc:654] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.608151 996 leader_election.cc:304] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:17.616694 1734 raft_consensus.cc:2798] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:17.616828 1734 raft_consensus.cc:491] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:17.616972 1734 raft_consensus.cc:3054] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.618069 521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_e55025c253874d759eae8dd088e59196"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:17.618372 1678 consensus_queue.cc:1035] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:17.618634 521 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e55025c253874d759eae8dd088e59196 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.621417 1178 ts_tablet_manager.cc:1918] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.621548 1178 log.cc:1198] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2e10e7654cec495bb532fb00e73c020d
I20250110 02:51:17.621893 1178 ts_tablet_manager.cc:1939] T 2e10e7654cec495bb532fb00e73c020d P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:17.622481 510 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2e10e7654cec495bb532fb00e73c020d (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]) successfully deleted
I20250110 02:51:17.633894 524 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_8d7fe23e0afe4dfdac17778b492f2c55 [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 3 got 0
I20250110 02:51:17.634658 1184 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.634842 1184 ts_tablet_manager.cc:1397] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250110 02:51:17.635686 1180 ts_tablet_manager.cc:1939] T 3fa76357abae4a95b8fb3af47288e43a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:17.635638 1184 raft_consensus.cc:357] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.635887 1184 raft_consensus.cc:383] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.635973 1184 raft_consensus.cc:738] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.636250 1184 consensus_queue.cc:260] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.636741 1184 ts_tablet_manager.cc:1428] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:17.637071 1184 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.638005 1734 raft_consensus.cc:513] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.639050 1184 tablet_bootstrap.cc:654] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.639130 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:17.639189 1734 leader_election.cc:290] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:17.639648 994 leader_election.cc:343] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.639895 1188 tablet_bootstrap.cc:492] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:17.640043 1188 ts_tablet_manager.cc:1397] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.300s user 0.004s sys 0.007s
I20250110 02:51:17.642320 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:17.642532 774 raft_consensus.cc:3054] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.645231 774 raft_consensus.cc:2463] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:17.646747 1181 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:17.651330 1184 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.651470 1184 ts_tablet_manager.cc:1397] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250110 02:51:17.652179 1184 raft_consensus.cc:357] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.652359 1184 raft_consensus.cc:383] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.652469 1184 raft_consensus.cc:738] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.652706 1184 consensus_queue.cc:260] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.653306 1184 ts_tablet_manager.cc:1428] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250110 02:51:17.653587 1184 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.655222 1184 tablet_bootstrap.cc:654] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.655668 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec427cc101bd49f6b9d1743e3eeebbd1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:17.655864 776 raft_consensus.cc:3054] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:17.656105 518 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46680:
name: "default.Table_34ce8eea83364d1ca7d8cfa100115b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:17.656606 518 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34ce8eea83364d1ca7d8cfa100115b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:17.657644 524 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_108282f5803c488098a01d20b1e4f31a" } modify_external_catalogs: true
I20250110 02:51:17.657807 524 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_108282f5803c488098a01d20b1e4f31a" } modify_external_catalogs: true
I20250110 02:51:17.659055 776 raft_consensus.cc:2463] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:17.659617 1588 raft_consensus.cc:491] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.659811 1588 raft_consensus.cc:513] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.660225 1188 raft_consensus.cc:357] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.660410 1188 raft_consensus.cc:383] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.660414 1588 leader_election.cc:290] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.660465 1188 raft_consensus.cc:738] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.660653 1188 consensus_queue.cc:260] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.660982 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:17.661437 709 leader_election.cc:343] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.664125 1184 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.664259 1184 ts_tablet_manager.cc:1397] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250110 02:51:17.661746 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:17.664892 711 leader_election.cc:343] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.665002 711 leader_election.cc:304] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.665066 1184 raft_consensus.cc:357] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.665256 1184 raft_consensus.cc:383] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.665331 1184 raft_consensus.cc:738] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.665386 1588 raft_consensus.cc:2743] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.665568 1184 consensus_queue.cc:260] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.665825 1181 ts_tablet_manager.cc:1397] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.526s user 0.047s sys 0.048s
I20250110 02:51:17.666208 1184 ts_tablet_manager.cc:1428] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:17.666977 1184 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.668000 512 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 3fa76357abae4a95b8fb3af47288e43a (table default.Table_dce75d4fa88e4bf59fe18cf4725d953f [id=7c8bb5b8a91a4ed9a81403e2e6196406]) successfully deleted
I20250110 02:51:17.668613 1184 tablet_bootstrap.cc:654] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:17.668706 512 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
I20250110 02:51:17.669168 1180 tablet_replica.cc:331] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:17.669421 1180 raft_consensus.cc:2238] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.669513 517 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e4c35b6a442b4b95a1341303de60226f" } modify_external_catalogs: true
I20250110 02:51:17.669665 517 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_e4c35b6a442b4b95a1341303de60226f" } modify_external_catalogs: true
I20250110 02:51:17.669981 1180 raft_consensus.cc:2267] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.670519 1180 ts_tablet_manager.cc:1905] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.665247 1747 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:17.663767 996 leader_election.cc:304] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:17.672376 996 leader_election.cc:304] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:17.672609 1678 raft_consensus.cc:2798] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:17.673024 1678 raft_consensus.cc:695] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:17.673291 1678 consensus_queue.cc:237] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.676965 519 master_service.cc:946] ReplaceTablet: received request to replace tablet f63d69d04b494a848653636c9e259105 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:17.679688 1680 raft_consensus.cc:2798] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:17.684554 1181 raft_consensus.cc:357] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.689117 1188 ts_tablet_manager.cc:1428] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250110 02:51:17.691399 1188 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:17.698539 1748 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250110 02:51:17.707122 522 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 815767a4c7624852bfedab9fa6cb5f44 consensus state has no leader
I20250110 02:51:17.714705 522 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9cd2f808ffbc4122be3501fc1a25c768" } modify_external_catalogs: true
I20250110 02:51:17.714923 522 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_9cd2f808ffbc4122be3501fc1a25c768" } modify_external_catalogs: true
I20250110 02:51:17.716951 1755 tablet.cc:1721] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:17.717850 1680 raft_consensus.cc:695] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:17.718755 1181 raft_consensus.cc:383] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.718420 1680 consensus_queue.cc:237] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.726727 1181 raft_consensus.cc:738] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.731453 1181 consensus_queue.cc:260] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.732864 1181 ts_tablet_manager.cc:1428] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20250110 02:51:17.734606 1588 raft_consensus.cc:491] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.734764 1588 raft_consensus.cc:513] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.735422 1588 leader_election.cc:290] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.759708 1748 tablet.cc:1716] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Skipping requested alter to schema version 3, tablet already version 3
I20250110 02:51:17.736420 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:17.738835 1747 tablet.cc:1716] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Skipping requested alter to schema version 3, tablet already version 3
I20250110 02:51:17.742060 1188 tablet_bootstrap.cc:654] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:17.765230 709 leader_election.cc:343] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.769449 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskadt4RW/build/asan/bin/kudu pid 494
I20250110 02:51:17.769822 1747 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250110 02:51:17.770210 525 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e554d00d05b4fa7a14c9b6159f9d726 consensus state has no leader
I20250110 02:51:17.771360 525 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:17.771924 525 master_service.cc:946] ReplaceTablet: received request to replace tablet 65ae1c19d2dc4778a1cebddbd505f907 from {username='slave'} at 127.0.0.1:46680
I20250110 02:51:17.772403 525 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_94cf6a20190647dd87d3616b069b0243" } modify_external_catalogs: true
I20250110 02:51:17.772584 525 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46680:
table { table_name: "default.Table_94cf6a20190647dd87d3616b069b0243" } modify_external_catalogs: true
I20250110 02:51:17.776710 1748 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:17.782611 1188 tablet_bootstrap.cc:492] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:17.782820 1188 ts_tablet_manager.cc:1397] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.000s
I20250110 02:51:17.783733 1188 raft_consensus.cc:357] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.783975 1188 raft_consensus.cc:383] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.784070 1188 raft_consensus.cc:738] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.785427 526 catalog_manager.cc:5526] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:17.785857 526 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
W20250110 02:51:17.786626 523 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3858638b6fc48d0bb8b8f97370a791d consensus state has no leader
I20250110 02:51:17.787822 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:17.788044 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_26b5c54688724d26b6bf4f3587158e9c [id=9e880e06a6d648b7bc44b651084a14a3]). Expected version 4 got 0
I20250110 02:51:17.788204 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09]). Expected version 3 got 0
I20250110 02:51:17.788379 520 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:17.794778 746 tablet_service.cc:1467] Processing CreateTablet for tablet bd238fa1922d456789b59bf5cee52327 (DEFAULT_TABLE table=default.Table_09cf33925be04c35b8a1304633c52217 [id=6f90cd9154c3419b8d3fd88395270251]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.795163 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd238fa1922d456789b59bf5cee52327. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.795195 1188 consensus_queue.cc:260] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.795781 1188 ts_tablet_manager.cc:1428] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250110 02:51:17.796061 1188 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:17.797734 1188 tablet_bootstrap.cc:654] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.798239 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8b255096800d4cd58324e2c06ff1a44c (DEFAULT_TABLE table=default.Table_c0c89321da6c41cbbf765c7562f60e95 [id=866547595c5d4cce9221971e917542eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.798611 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b255096800d4cd58324e2c06ff1a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.801231 1184 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:17.801355 1184 ts_tablet_manager.cc:1397] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.024s
I20250110 02:51:17.802510 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.802788 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:55720
I20250110 02:51:17.803166 753 tablet_service.cc:1467] Processing CreateTablet for tablet ff57f894a5b54dee91452cbb4cb38b0c (DEFAULT_TABLE table=default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e [id=6eaaca051e6c496eaef0567ca4b21030]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.803483 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff57f894a5b54dee91452cbb4cb38b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.805776 1184 raft_consensus.cc:357] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.805951 1184 raft_consensus.cc:383] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:17.806028 1184 raft_consensus.cc:738] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:17.806237 1184 consensus_queue.cc:260] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.806660 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b (DEFAULT_TABLE table=default.Table_97ca49cb31634f579d9d84e0d4fc8057 [id=6acef2fac29441ceac28a2721295136a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.807049 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78a9ef5eb443da0c8983a7190440b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.809391 1184 ts_tablet_manager.cc:1428] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:17.809664 1184 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:17.810292 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1d634dd0c2044f91b304fe3737330d88 (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.810667 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d634dd0c2044f91b304fe3737330d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.811342 1184 tablet_bootstrap.cc:654] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.814678 741 tablet_service.cc:1467] Processing CreateTablet for tablet 340882f7684a4e7c9a95800117fc758b (DEFAULT_TABLE table=default.Table_2e1a57f017f844aba03084fd604240cf [id=cdcb9c6db16b4d11ade1bd993b0756fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.831221 881 tablet_service.cc:1467] Processing CreateTablet for tablet 7752241af2fb4f959f00252faee87e98 (DEFAULT_TABLE table=default.Table_410dc5667b5446fca70ebd92193bbc32 [id=1d5bb5a0ce4e4f91babd89611af071ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.831359 883 tablet_service.cc:1467] Processing CreateTablet for tablet e55ee565b7f14936a2cb4cbed716c153 (DEFAULT_TABLE table=default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.831672 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7752241af2fb4f959f00252faee87e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.831840 884 tablet_service.cc:1467] Processing CreateTablet for tablet bd238fa1922d456789b59bf5cee52327 (DEFAULT_TABLE table=default.Table_09cf33925be04c35b8a1304633c52217 [id=6f90cd9154c3419b8d3fd88395270251]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.832001 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 (DEFAULT_TABLE table=default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.832172 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd238fa1922d456789b59bf5cee52327. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.832470 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1f2acd39be40e4b71cc0dcf68e7567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.832630 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8b255096800d4cd58324e2c06ff1a44c (DEFAULT_TABLE table=default.Table_c0c89321da6c41cbbf765c7562f60e95 [id=866547595c5d4cce9221971e917542eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.832962 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b255096800d4cd58324e2c06ff1a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.834704 879 tablet_service.cc:1467] Processing CreateTablet for tablet ff57f894a5b54dee91452cbb4cb38b0c (DEFAULT_TABLE table=default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e [id=6eaaca051e6c496eaef0567ca4b21030]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.835072 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff57f894a5b54dee91452cbb4cb38b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.838699 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.838959 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1d634dd0c2044f91b304fe3737330d88 (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.839141 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb805b3e537466ab7fe0185421ad5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.839347 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d634dd0c2044f91b304fe3737330d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.838725 886 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b (DEFAULT_TABLE table=default.Table_97ca49cb31634f579d9d84e0d4fc8057 [id=6acef2fac29441ceac28a2721295136a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.839855 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78a9ef5eb443da0c8983a7190440b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.842496 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55ee565b7f14936a2cb4cbed716c153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.847101 892 tablet_service.cc:1467] Processing CreateTablet for tablet bc8444c1381a45d6b2fee30da971bd47 (DEFAULT_TABLE table=default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.847770 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8600a637306741dd8e1e7f4964ff4bc8 (DEFAULT_TABLE table=default.Table_eb40720cee0c4ef08e2c5a49f1374c8e [id=40f4c299cde64d02bd2a420d11b6765f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.848423 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2da1ee1add01482cbf651bd2b4753dc1 (DEFAULT_TABLE table=default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 [id=722b4aa4edcb4752a0df9a714e558255]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.848870 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340882f7684a4e7c9a95800117fc758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.849107 1588 raft_consensus.cc:491] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.849261 1588 raft_consensus.cc:513] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.849550 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7752241af2fb4f959f00252faee87e98 (DEFAULT_TABLE table=default.Table_410dc5667b5446fca70ebd92193bbc32 [id=1d5bb5a0ce4e4f91babd89611af071ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.850200 755 tablet_service.cc:1467] Processing CreateTablet for tablet e55ee565b7f14936a2cb4cbed716c153 (DEFAULT_TABLE table=default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.855538 896 tablet_service.cc:1467] Processing CreateTablet for tablet deaa3527aa074e6ba94960c672aee028 (DEFAULT_TABLE table=default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.856074 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8600a637306741dd8e1e7f4964ff4bc8 (DEFAULT_TABLE table=default.Table_eb40720cee0c4ef08e2c5a49f1374c8e [id=40f4c299cde64d02bd2a420d11b6765f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.856369 898 tablet_service.cc:1467] Processing CreateTablet for tablet 2da1ee1add01482cbf651bd2b4753dc1 (DEFAULT_TABLE table=default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 [id=722b4aa4edcb4752a0df9a714e558255]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.856637 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1253c54d63cc47f081e676851305bc4b (DEFAULT_TABLE table=default.Table_8a8d43a5e45c47f083ed451f5b30bf6b [id=2387784c829c4e8bb0ecc7d63bb42111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.856909 887 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb775de0154409a0d92990a51d1ee0 (DEFAULT_TABLE table=default.Table_35002d6ab8564fe1b93188d7ce973d3f [id=ed3a9f13f7794c3f9ee6917bf6ecff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.857208 885 tablet_service.cc:1467] Processing CreateTablet for tablet fc1324eeac9446279304f442e2fad488 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.858769 880 tablet_service.cc:1467] Processing CreateTablet for tablet 340882f7684a4e7c9a95800117fc758b (DEFAULT_TABLE table=default.Table_2e1a57f017f844aba03084fd604240cf [id=cdcb9c6db16b4d11ade1bd993b0756fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.859203 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8444c1381a45d6b2fee30da971bd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.864342 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb775de0154409a0d92990a51d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.865087 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da1ee1add01482cbf651bd2b4753dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.866773 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1324eeac9446279304f442e2fad488. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.867780 1179 tablet_replica.cc:331] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.868124 1716 raft_consensus.cc:491] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.868278 1716 raft_consensus.cc:513] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:17.869653 743 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb775de0154409a0d92990a51d1ee0 (DEFAULT_TABLE table=default.Table_35002d6ab8564fe1b93188d7ce973d3f [id=ed3a9f13f7794c3f9ee6917bf6ecff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.870319 752 tablet_service.cc:1467] Processing CreateTablet for tablet fc1324eeac9446279304f442e2fad488 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.870952 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.873836 747 tablet_service.cc:1467] Processing CreateTablet for tablet deaa3527aa074e6ba94960c672aee028 (DEFAULT_TABLE table=default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.875125 750 tablet_service.cc:1467] Processing CreateTablet for tablet bc8444c1381a45d6b2fee30da971bd47 (DEFAULT_TABLE table=default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.878198 1179 raft_consensus.cc:2238] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.878862 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1253c54d63cc47f081e676851305bc4b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:17.878017 853 leader_election.cc:343] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:17.880206 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1253c54d63cc47f081e676851305bc4b (DEFAULT_TABLE table=default.Table_8a8d43a5e45c47f083ed451f5b30bf6b [id=2387784c829c4e8bb0ecc7d63bb42111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.880333 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deaa3527aa074e6ba94960c672aee028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.880525 1716 leader_election.cc:290] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.881182 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8600a637306741dd8e1e7f4964ff4bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.881373 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deaa3527aa074e6ba94960c672aee028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.882071 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 (DEFAULT_TABLE table=default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.882432 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1253c54d63cc47f081e676851305bc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.883284 1181 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:17.883317 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.884640 1179 raft_consensus.cc:2267] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.885133 1181 tablet_bootstrap.cc:654] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:17.886036 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340882f7684a4e7c9a95800117fc758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.886117 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55ee565b7f14936a2cb4cbed716c153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.886623 1760 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:17.886809 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da1ee1add01482cbf651bd2b4753dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.887665 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb805b3e537466ab7fe0185421ad5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.888608 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1324eeac9446279304f442e2fad488. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.889434 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb775de0154409a0d92990a51d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.890347 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8444c1381a45d6b2fee30da971bd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.891243 1588 leader_election.cc:290] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.891721 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7752241af2fb4f959f00252faee87e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.892309 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8600a637306741dd8e1e7f4964ff4bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.892791 1588 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:17.892848 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 8600a637306741dd8e1e7f4964ff4bc8 (DEFAULT_TABLE table=default.Table_eb40720cee0c4ef08e2c5a49f1374c8e [id=40f4c299cde64d02bd2a420d11b6765f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.893265 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8600a637306741dd8e1e7f4964ff4bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.893225 1588 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:17.893625 1179 ts_tablet_manager.cc:1905] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.894279 1588 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:17.894605 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1f2acd39be40e4b71cc0dcf68e7567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.895606 1033 tablet_service.cc:1467] Processing CreateTablet for tablet 2da1ee1add01482cbf651bd2b4753dc1 (DEFAULT_TABLE table=default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 [id=722b4aa4edcb4752a0df9a714e558255]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.895982 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da1ee1add01482cbf651bd2b4753dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.897289 1678 consensus_queue.cc:1035] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:17.898680 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 (DEFAULT_TABLE table=default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.899011 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1f2acd39be40e4b71cc0dcf68e7567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.902427 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1d634dd0c2044f91b304fe3737330d88 (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.902721 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d634dd0c2044f91b304fe3737330d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.906438 1032 tablet_service.cc:1467] Processing CreateTablet for tablet ff57f894a5b54dee91452cbb4cb38b0c (DEFAULT_TABLE table=default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e [id=6eaaca051e6c496eaef0567ca4b21030]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.906793 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff57f894a5b54dee91452cbb4cb38b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.913642 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bc8444c1381a45d6b2fee30da971bd47 (DEFAULT_TABLE table=default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.887795 1752 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:17.916229 1038 tablet_service.cc:1467] Processing CreateTablet for tablet bd238fa1922d456789b59bf5cee52327 (DEFAULT_TABLE table=default.Table_09cf33925be04c35b8a1304633c52217 [id=6f90cd9154c3419b8d3fd88395270251]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.917244 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b (DEFAULT_TABLE table=default.Table_97ca49cb31634f579d9d84e0d4fc8057 [id=6acef2fac29441ceac28a2721295136a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.915562 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 340882f7684a4e7c9a95800117fc758b (DEFAULT_TABLE table=default.Table_2e1a57f017f844aba03084fd604240cf [id=cdcb9c6db16b4d11ade1bd993b0756fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.924739 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 7752241af2fb4f959f00252faee87e98 (DEFAULT_TABLE table=default.Table_410dc5667b5446fca70ebd92193bbc32 [id=1d5bb5a0ce4e4f91babd89611af071ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.924947 1036 tablet_service.cc:1467] Processing CreateTablet for tablet e55ee565b7f14936a2cb4cbed716c153 (DEFAULT_TABLE table=default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.917420 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8b255096800d4cd58324e2c06ff1a44c (DEFAULT_TABLE table=default.Table_c0c89321da6c41cbbf765c7562f60e95 [id=866547595c5d4cce9221971e917542eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.935428 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8444c1381a45d6b2fee30da971bd47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.957877 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 02:51:17.968299 1039 tablet_service.cc:1467] Processing CreateTablet for tablet deaa3527aa074e6ba94960c672aee028 (DEFAULT_TABLE table=default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.968143 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1253c54d63cc47f081e676851305bc4b (DEFAULT_TABLE table=default.Table_8a8d43a5e45c47f083ed451f5b30bf6b [id=2387784c829c4e8bb0ecc7d63bb42111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.970801 1037 tablet_service.cc:1467] Processing CreateTablet for tablet fc1324eeac9446279304f442e2fad488 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:17.971015 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 247556us
I20250110 02:51:17.970798 1035 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb775de0154409a0d92990a51d1ee0 (DEFAULT_TABLE table=default.Table_35002d6ab8564fe1b93188d7ce973d3f [id=ed3a9f13f7794c3f9ee6917bf6ecff25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:17.972951 1747 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:17.975672 1179 ts_tablet_manager.cc:1918] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.975786 1179 log.cc:1198] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:17.978525 1179 ts_tablet_manager.cc:1939] T 09f01daf4c67462ba75e1d0cf56cef12 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.979591 512 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 09f01daf4c67462ba75e1d0cf56cef12 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
I20250110 02:51:17.979784 1748 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:17.979907 1179 tablet_replica.cc:331] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:17.980127 1179 raft_consensus.cc:2238] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:17.980399 1179 raft_consensus.cc:2267] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:17.981127 1179 ts_tablet_manager.cc:1905] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:17.982750 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78a9ef5eb443da0c8983a7190440b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:17.984915 711 leader_election.cc:343] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:17.985046 711 leader_election.cc:304] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:17.985291 1588 raft_consensus.cc:2743] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:17.987877 1179 ts_tablet_manager.cc:1918] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:17.988044 1179 log.cc:1198] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8c3ce95413df40369ee599176f91b892
I20250110 02:51:17.988368 1179 ts_tablet_manager.cc:1939] T 8c3ce95413df40369ee599176f91b892 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:17.993072 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deaa3527aa074e6ba94960c672aee028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.994330 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:17.995390 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:17.999732 853 leader_election.cc:343] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:17.999877 853 leader_election.cc:304] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.000268 1716 raft_consensus.cc:2743] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:18.000859 711 leader_election.cc:343] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.001075 711 leader_election.cc:343] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.001577 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b255096800d4cd58324e2c06ff1a44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:17.999521 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:18.006268 1678 consensus_queue.cc:1035] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250110 02:51:18.010442 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340882f7684a4e7c9a95800117fc758b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.010659 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1324eeac9446279304f442e2fad488. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.010859 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb805b3e537466ab7fe0185421ad5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.011873 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1253c54d63cc47f081e676851305bc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.012408 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb775de0154409a0d92990a51d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:18.016640 994 consensus_peers.cc:487] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:18.024416 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7752241af2fb4f959f00252faee87e98. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:18.027753 1755 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.263s user 0.000s sys 0.000s
I20250110 02:51:18.033895 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55ee565b7f14936a2cb4cbed716c153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.037294 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.039216 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.040383 709 leader_election.cc:343] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.040530 709 leader_election.cc:304] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.040812 1768 raft_consensus.cc:2743] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:18.041152 709 leader_election.cc:343] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.041253 709 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.041497 1768 raft_consensus.cc:2743] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.082095 1748 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250110 02:51:18.083770 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd238fa1922d456789b59bf5cee52327. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:18.091702 1755 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250110 02:51:18.096930 1752 tablet.cc:1716] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Skipping requested alter to schema version 3, tablet already version 3
I20250110 02:51:18.101855 1768 raft_consensus.cc:491] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.102082 1768 raft_consensus.cc:513] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.103698 1768 leader_election.cc:290] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.104311 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.104988 709 leader_election.cc:343] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.140357 1760 tablet.cc:1721] T c09235548b5d4e10a01cbe05d1c94841 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:18.142616 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.143481 711 leader_election.cc:343] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.143668 711 leader_election.cc:304] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.144013 1768 raft_consensus.cc:2743] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.160848 1188 tablet_bootstrap.cc:492] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.161121 1188 ts_tablet_manager.cc:1397] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.365s user 0.030s sys 0.066s
I20250110 02:51:18.161978 1188 raft_consensus.cc:357] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.162292 1188 raft_consensus.cc:383] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.162436 1188 raft_consensus.cc:738] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.162777 1188 consensus_queue.cc:260] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.167630 1188 ts_tablet_manager.cc:1428] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250110 02:51:18.169986 1184 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.171365 1188 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.173323 1188 tablet_bootstrap.cc:654] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.181663 1188 tablet_bootstrap.cc:492] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.181891 1188 ts_tablet_manager.cc:1397] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20250110 02:51:18.182830 1188 raft_consensus.cc:357] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.183105 1188 raft_consensus.cc:383] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.183231 1188 raft_consensus.cc:738] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.183502 1188 consensus_queue.cc:260] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.184916 1180 ts_tablet_manager.cc:1918] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:18.185716 1184 ts_tablet_manager.cc:1397] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.376s user 0.009s sys 0.008s
I20250110 02:51:18.187686 1188 ts_tablet_manager.cc:1428] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:18.191592 1188 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.193324 1188 tablet_bootstrap.cc:654] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.198834 1184 raft_consensus.cc:357] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.199215 1184 raft_consensus.cc:383] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.199369 1184 raft_consensus.cc:738] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.199774 1184 consensus_queue.cc:260] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.200493 1184 ts_tablet_manager.cc:1428] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250110 02:51:18.202665 1188 tablet_bootstrap.cc:492] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.203339 1188 ts_tablet_manager.cc:1397] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
W20250110 02:51:18.203804 994 consensus_peers.cc:487] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:18.204056 1188 raft_consensus.cc:357] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.204264 1188 raft_consensus.cc:383] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.204347 1188 raft_consensus.cc:738] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.204654 1188 consensus_queue.cc:260] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.207581 1180 log.cc:1198] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/344184e1c98d422095cb254f5b2bc792
I20250110 02:51:18.208098 1180 ts_tablet_manager.cc:1939] T 344184e1c98d422095cb254f5b2bc792 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:18.209105 1188 ts_tablet_manager.cc:1428] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:18.212648 1188 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.213718 1184 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.231302 1184 tablet_bootstrap.cc:654] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.241739 1188 tablet_bootstrap.cc:654] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.250053 1188 tablet_bootstrap.cc:492] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.250285 1188 ts_tablet_manager.cc:1397] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.005s
I20250110 02:51:18.251082 1188 raft_consensus.cc:357] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.251349 1188 raft_consensus.cc:383] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.251470 1188 raft_consensus.cc:738] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.251780 1188 consensus_queue.cc:260] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.256445 1188 ts_tablet_manager.cc:1428] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:18.260493 1188 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.260999 1181 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.261198 1181 ts_tablet_manager.cc:1397] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.528s user 0.005s sys 0.011s
I20250110 02:51:18.261526 1726 raft_consensus.cc:491] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.261674 1726 raft_consensus.cc:513] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.262087 1184 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.262037 1181 raft_consensus.cc:357] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.262208 1181 raft_consensus.cc:383] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.262212 1184 ts_tablet_manager.cc:1397] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250110 02:51:18.262233 1726 leader_election.cc:290] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.262284 1181 raft_consensus.cc:738] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.263005 1184 raft_consensus.cc:357] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.263314 1184 raft_consensus.cc:383] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.263439 1184 raft_consensus.cc:738] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.263793 1184 consensus_queue.cc:260] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:18.263998 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:18.264463 1184 ts_tablet_manager.cc:1428] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250110 02:51:18.264518 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:18.264632 853 leader_election.cc:304] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.264765 1184 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.265000 1726 raft_consensus.cc:2743] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.265122 1188 tablet_bootstrap.cc:654] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.269220 1181 consensus_queue.cc:260] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.269721 1181 ts_tablet_manager.cc:1428] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250110 02:51:18.271106 1181 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.272557 1181 tablet_bootstrap.cc:654] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.284026 1184 tablet_bootstrap.cc:654] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.300179 1181 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.300441 1181 ts_tablet_manager.cc:1397] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250110 02:51:18.301407 1181 raft_consensus.cc:357] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.301808 1181 raft_consensus.cc:383] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.301929 1181 raft_consensus.cc:738] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.302393 1759 raft_consensus.cc:491] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.302695 1759 raft_consensus.cc:513] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.303071 1184 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.303289 1184 ts_tablet_manager.cc:1397] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250110 02:51:18.303905 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.304108 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.304363 774 raft_consensus.cc:2463] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:18.304422 994 leader_election.cc:343] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.304955 996 leader_election.cc:304] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:18.305006 1184 raft_consensus.cc:357] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.305194 1184 raft_consensus.cc:383] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.305243 1184 raft_consensus.cc:738] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.305287 1770 raft_consensus.cc:2798] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:18.305375 1770 raft_consensus.cc:491] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:18.305477 1770 raft_consensus.cc:3054] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.305424 1184 consensus_queue.cc:260] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.305850 1184 ts_tablet_manager.cc:1428] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:18.306102 1184 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.314088 1770 raft_consensus.cc:513] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.319268 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:18.320015 994 leader_election.cc:343] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.320169 1184 tablet_bootstrap.cc:654] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.321261 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:18.321507 774 raft_consensus.cc:3054] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.325829 774 raft_consensus.cc:2463] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:18.326831 1770 leader_election.cc:290] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.303598 1759 leader_election.cc:290] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.327014 996 leader_election.cc:304] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:18.327335 1188 tablet_bootstrap.cc:492] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.327365 1759 raft_consensus.cc:2798] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:18.327487 1188 ts_tablet_manager.cc:1397] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.067s user 0.032s sys 0.030s
I20250110 02:51:18.327960 1759 raft_consensus.cc:695] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:18.328347 1188 raft_consensus.cc:357] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.328369 1759 consensus_queue.cc:237] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.328596 1188 raft_consensus.cc:383] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.328662 1188 raft_consensus.cc:738] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.328891 1188 consensus_queue.cc:260] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.329381 1188 ts_tablet_manager.cc:1428] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:18.329727 1188 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.331061 1760 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250110 02:51:18.331846 1188 tablet_bootstrap.cc:654] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.331950 1181 consensus_queue.cc:260] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.332602 1181 ts_tablet_manager.cc:1428] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250110 02:51:18.333462 1184 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.333587 1184 ts_tablet_manager.cc:1397] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250110 02:51:18.333886 1181 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.334220 1184 raft_consensus.cc:357] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.334422 1184 raft_consensus.cc:383] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.334497 1184 raft_consensus.cc:738] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.334671 1184 consensus_queue.cc:260] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.335105 1184 ts_tablet_manager.cc:1428] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 02:51:18.335328 1184 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.336691 1184 tablet_bootstrap.cc:654] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.336696 1181 tablet_bootstrap.cc:654] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.339932 1789 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8bc934cc708045e9858ee8f8e64660cd)
I20250110 02:51:18.351058 1789 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:18.351981 1184 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.352152 1184 ts_tablet_manager.cc:1397] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250110 02:51:18.353613 1789 leader_election.cc:290] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:18.352803 1184 raft_consensus.cc:357] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.354176 1184 raft_consensus.cc:383] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.354286 1184 raft_consensus.cc:738] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.354252 1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 4 candidate_status { last_received { term: 3 index: 646 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:18.354646 1184 consensus_queue.cc:260] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.361434 1184 ts_tablet_manager.cc:1428] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250110 02:51:18.364996 1184 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.368695 1184 tablet_bootstrap.cc:654] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.369164 1181 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.369367 1181 ts_tablet_manager.cc:1397] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250110 02:51:18.375641 1181 raft_consensus.cc:357] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.375988 1188 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.376214 1188 ts_tablet_manager.cc:1397] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.005s
I20250110 02:51:18.376343 1181 raft_consensus.cc:383] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.376643 1181 raft_consensus.cc:738] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.377147 1188 raft_consensus.cc:357] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.377593 1188 raft_consensus.cc:383] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.377223 1181 consensus_queue.cc:260] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.377758 1188 raft_consensus.cc:738] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.378049 1188 consensus_queue.cc:260] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.378765 1188 ts_tablet_manager.cc:1428] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:18.380636 1181 ts_tablet_manager.cc:1428] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250110 02:51:18.381300 1188 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.383172 1188 tablet_bootstrap.cc:654] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.383337 1768 raft_consensus.cc:491] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.383474 1768 raft_consensus.cc:513] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.384071 1768 leader_election.cc:290] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.386925 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.387547 711 leader_election.cc:343] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.388572 1181 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.385058 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.389511 709 leader_election.cc:343] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.389662 709 leader_election.cc:304] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.390460 1181 tablet_bootstrap.cc:654] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.390616 1768 raft_consensus.cc:2743] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.399127 1795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8bc934cc708045e9858ee8f8e64660cd)
I20250110 02:51:18.399482 1795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:18.400523 1188 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.400753 1188 ts_tablet_manager.cc:1397] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250110 02:51:18.401510 1188 raft_consensus.cc:357] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.401765 1188 raft_consensus.cc:383] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.401863 1188 raft_consensus.cc:738] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.402128 1188 consensus_queue.cc:260] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.402694 1188 ts_tablet_manager.cc:1428] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:18.403024 1188 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.405196 1188 tablet_bootstrap.cc:654] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.411026 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 646 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:18.411235 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d8d40556dc348e8aa8255d116251a10 in term 3.
I20250110 02:51:18.411660 1433 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters:
I20250110 02:51:18.418669 1795 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
I20250110 02:51:18.419173 1795 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250110 02:51:18.419714 1795 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8bc934cc708045e9858ee8f8e64660cd)
I20250110 02:51:18.420150 1795 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 3 FOLLOWER]: Advancing to term 4
I20250110 02:51:18.428987 1188 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.429220 1188 ts_tablet_manager.cc:1397] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250110 02:51:18.430539 1188 raft_consensus.cc:357] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.430816 1188 raft_consensus.cc:383] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.430933 1188 raft_consensus.cc:738] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.431206 1188 consensus_queue.cc:260] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.431774 1188 ts_tablet_manager.cc:1428] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:18.438654 1188 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.440454 1188 tablet_bootstrap.cc:654] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.454289 1726 raft_consensus.cc:491] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.454566 1726 raft_consensus.cc:513] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.455389 1726 leader_election.cc:290] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.456476 1768 raft_consensus.cc:491] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.456580 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.456647 1768 raft_consensus.cc:513] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.457289 1768 leader_election.cc:290] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:18.457717 1433 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:35975, user_credentials={real_user=slave}} blocked reactor thread for 46082.1us
I20250110 02:51:18.459702 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.460263 709 leader_election.cc:343] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.460580 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934e6611d4e4b94ba5185f33c3d05f4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.460806 774 raft_consensus.cc:2388] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:18.461215 853 leader_election.cc:304] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.461570 1726 raft_consensus.cc:3054] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.464359 1726 raft_consensus.cc:2743] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.465091 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.465611 711 leader_election.cc:343] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.465740 711 leader_election.cc:304] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.466179 1768 raft_consensus.cc:2743] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.468735 1795 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:18.470340 1181 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.470630 1181 ts_tablet_manager.cc:1397] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.005s
I20250110 02:51:18.471714 1181 raft_consensus.cc:357] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.472091 1181 raft_consensus.cc:383] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.472226 1181 raft_consensus.cc:738] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.472616 1181 consensus_queue.cc:260] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.474916 1181 ts_tablet_manager.cc:1428] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:18.475240 1181 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.481058 1181 tablet_bootstrap.cc:654] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.483928 1184 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.484164 1184 ts_tablet_manager.cc:1397] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.119s user 0.022s sys 0.054s
I20250110 02:51:18.485913 1184 raft_consensus.cc:357] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.486213 1184 raft_consensus.cc:383] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.486340 1184 raft_consensus.cc:738] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.486672 1184 consensus_queue.cc:260] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.492705 1184 ts_tablet_manager.cc:1428] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:18.496270 1184 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.498039 1184 tablet_bootstrap.cc:654] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.506115 1184 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.506337 1184 ts_tablet_manager.cc:1397] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20250110 02:51:18.507128 1184 raft_consensus.cc:357] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.507424 1184 raft_consensus.cc:383] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.507536 1184 raft_consensus.cc:738] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.507850 1184 consensus_queue.cc:260] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.512015 1184 ts_tablet_manager.cc:1428] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:18.516741 1184 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.517282 1726 raft_consensus.cc:491] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.517444 1726 raft_consensus.cc:513] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.518272 1726 leader_election.cc:290] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:18.519668 853 leader_election.cc:343] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 019c0d60da5548bb86c93ccd89bf94ff
W20250110 02:51:18.519915 853 leader_election.cc:343] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:18.520030 853 leader_election.cc:304] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.520433 1726 raft_consensus.cc:2743] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.521917 1184 tablet_bootstrap.cc:654] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.523241 1795 leader_election.cc:290] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 election: Requested vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449)
I20250110 02:51:18.523280 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 646 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558"
I20250110 02:51:18.523573 465 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 3 FOLLOWER]: Advancing to term 4
I20250110 02:51:18.527086 465 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d8d40556dc348e8aa8255d116251a10 in term 4.
I20250110 02:51:18.527488 1433 leader_election.cc:304] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558; no voters:
I20250110 02:51:18.527869 1795 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 4 FOLLOWER]: Leader election won for term 4
I20250110 02:51:18.528234 1795 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [term 4 LEADER]: Becoming Leader. State: Replica: 5d8d40556dc348e8aa8255d116251a10, State: Running, Role: LEADER
I20250110 02:51:18.528812 1795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 646, Committed index: 646, Last appended: 3.646, Last appended by leader: 646, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:18.529982 1184 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.530149 1184 ts_tablet_manager.cc:1397] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20250110 02:51:18.530942 1184 raft_consensus.cc:357] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.530927 1798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8d40556dc348e8aa8255d116251a10. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:18.531128 1798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:18.531226 1184 raft_consensus.cc:383] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.531356 1184 raft_consensus.cc:738] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.531698 1184 consensus_queue.cc:260] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.539032 1181 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.539903 1188 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.540163 1188 ts_tablet_manager.cc:1397] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.102s user 0.022s sys 0.048s
I20250110 02:51:18.540970 1188 raft_consensus.cc:357] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.541286 1188 raft_consensus.cc:383] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.541404 1188 raft_consensus.cc:738] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.541535 1184 ts_tablet_manager.cc:1428] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:18.541759 1188 consensus_queue.cc:260] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.542308 1188 ts_tablet_manager.cc:1428] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:18.545848 1726 raft_consensus.cc:491] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.545989 1726 raft_consensus.cc:513] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.546586 1726 leader_election.cc:290] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:18.547966 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
W20250110 02:51:18.548172 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:18.548282 853 leader_election.cc:304] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.550238 1726 raft_consensus.cc:2743] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.552088 1184 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.554054 1184 tablet_bootstrap.cc:654] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.558447 1188 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.558763 1804 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 02:51:18.560556 1188 tablet_bootstrap.cc:654] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.560894 1181 ts_tablet_manager.cc:1397] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.003s
I20250110 02:51:18.562729 1181 raft_consensus.cc:357] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.562994 1181 raft_consensus.cc:383] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.563124 1181 raft_consensus.cc:738] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.563366 1726 raft_consensus.cc:491] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.563488 1726 raft_consensus.cc:513] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.564095 1726 leader_election.cc:290] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.564487 1181 consensus_queue.cc:260] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.564725 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.565089 1181 ts_tablet_manager.cc:1428] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:18.565174 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d4c4631994a5792200ba4d4c906f0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.566011 853 leader_election.cc:304] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.566442 1726 raft_consensus.cc:2743] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.572252 1181 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.575788 1181 tablet_bootstrap.cc:654] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.578207 1184 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.578483 1184 ts_tablet_manager.cc:1397] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250110 02:51:18.579555 1184 raft_consensus.cc:357] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.579809 1184 raft_consensus.cc:383] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.579903 1184 raft_consensus.cc:738] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.580195 1184 consensus_queue.cc:260] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.580714 1188 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.580885 1188 ts_tablet_manager.cc:1397] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20250110 02:51:18.581640 1188 raft_consensus.cc:357] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.581914 1188 raft_consensus.cc:383] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.582060 1188 raft_consensus.cc:738] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.582335 1188 consensus_queue.cc:260] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.582921 1188 ts_tablet_manager.cc:1428] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:18.587268 1184 ts_tablet_manager.cc:1428] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:18.590037 1726 raft_consensus.cc:491] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.590162 1726 raft_consensus.cc:513] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.590862 1726 leader_election.cc:290] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.591989 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af3d61366b2432db53b18f2b7bb49d9" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.592854 853 leader_election.cc:343] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.593066 853 leader_election.cc:343] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:18.593154 853 leader_election.cc:304] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.593514 1726 raft_consensus.cc:2743] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.595312 1188 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.599576 1184 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.601289 1184 tablet_bootstrap.cc:654] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.604442 1188 tablet_bootstrap.cc:654] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.605259 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]
I20250110 02:51:18.610548 1770 consensus_queue.cc:1035] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:18.622300 1181 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.622532 1181 ts_tablet_manager.cc:1397] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20250110 02:51:18.623000 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]
I20250110 02:51:18.623415 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]
I20250110 02:51:18.623466 1181 raft_consensus.cc:357] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.623726 1181 raft_consensus.cc:383] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.623811 1181 raft_consensus.cc:738] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.623840 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]
I20250110 02:51:18.624203 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 [id=056af6d890614cae954fa6f9e3f7f6d7]
I20250110 02:51:18.624107 1181 consensus_queue.cc:260] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.624552 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3]
I20250110 02:51:18.624689 1181 ts_tablet_manager.cc:1428] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:18.625123 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]
I20250110 02:51:18.625560 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]
I20250110 02:51:18.625830 1181 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.625980 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]
I20250110 02:51:18.626434 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]
I20250110 02:51:18.630976 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_73259c80bde24fd3a2440bdce73bd88e [id=0afe16dcfb0c4ef0897ceb5fb2993d7b]
I20250110 02:51:18.630985 1734 raft_consensus.cc:491] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.631156 1734 raft_consensus.cc:513] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.631451 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_27ec9682796f420ebd7920d531fce2fa [id=0b98c3d632294ec4bfa3fe55e718afac]
I20250110 02:51:18.631825 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]
I20250110 02:51:18.631830 1734 leader_election.cc:290] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.632383 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f9ee107db2ba4f2d9b62e9459ea41457 [id=0e2933f55f834e8baa0a11bda1b4acb4]
I20250110 02:51:18.632717 1184 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.632870 1184 ts_tablet_manager.cc:1397] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.003s
I20250110 02:51:18.633167 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c [id=103f9b0512f343a29e3d8ba6de6071ee]
I20250110 02:51:18.633499 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]
I20250110 02:51:18.633857 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253]
I20250110 02:51:18.633885 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.634120 774 raft_consensus.cc:2463] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:18.634573 1726 raft_consensus.cc:491] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.634763 996 leader_election.cc:304] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:18.634738 1726 raft_consensus.cc:513] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.635056 1680 consensus_queue.cc:1035] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:18.635346 1726 leader_election.cc:290] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.635402 1680 raft_consensus.cc:2798] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:18.635495 1680 raft_consensus.cc:491] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:18.635579 1680 raft_consensus.cc:3054] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:18.639780 853 leader_election.cc:343] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:18.641036 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_7fdb03687ec14267bfee2826ddb5a13b [id=14b73a8ae2744e1ea505735074b5c3ee]
I20250110 02:51:18.642196 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.642997 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a3ba2bc51a3b4fc6b6451349e08b106d [id=14dfd1504b024d6bbd154720e1e8c43f]
I20250110 02:51:18.638347 1680 raft_consensus.cc:513] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.639195 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e72ed2d31a8467f8c092b48d593c1cc" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.643435 853 leader_election.cc:343] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.643496 1680 leader_election.cc:290] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.643563 853 leader_election.cc:304] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.644074 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d]
I20250110 02:51:18.644763 1726 raft_consensus.cc:2743] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.645630 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]
I20250110 02:51:18.645526 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:18.645781 775 raft_consensus.cc:3054] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.646193 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]
I20250110 02:51:18.646798 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]
I20250110 02:51:18.647181 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]
I20250110 02:51:18.647548 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]
I20250110 02:51:18.647943 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_410dc5667b5446fca70ebd92193bbc32 [id=1d5bb5a0ce4e4f91babd89611af071ac]
I20250110 02:51:18.648962 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8a8d43a5e45c47f083ed451f5b30bf6b [id=2387784c829c4e8bb0ecc7d63bb42111]
I20250110 02:51:18.649091 775 raft_consensus.cc:2463] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:18.648619 1184 raft_consensus.cc:357] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.649215 1184 raft_consensus.cc:383] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.649300 1184 raft_consensus.cc:738] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.649338 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_febf1d5b7a274c5fb59e10a180b4dd59 [id=23e8c0cf343748a092d750ea2ff7647a]
I20250110 02:51:18.649544 1184 consensus_queue.cc:260] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.649729 996 leader_election.cc:304] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:18.650118 1184 ts_tablet_manager.cc:1428] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250110 02:51:18.650130 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9]
I20250110 02:51:18.650230 1680 raft_consensus.cc:2798] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:18.650427 1184 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.650756 1680 raft_consensus.cc:695] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:18.651147 1680 consensus_queue.cc:237] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:18.642771 994 leader_election.cc:343] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.642053 1181 tablet_bootstrap.cc:654] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.653321 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:18.653643 1184 tablet_bootstrap.cc:654] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:18.654675 994 leader_election.cc:343] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.655381 1188 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.655561 1188 ts_tablet_manager.cc:1397] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.004s
I20250110 02:51:18.657629 1188 raft_consensus.cc:357] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.657929 1188 raft_consensus.cc:383] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.657985 1188 raft_consensus.cc:738] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.662721 1188 consensus_queue.cc:260] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.664978 1188 ts_tablet_manager.cc:1428] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20250110 02:51:18.665268 1188 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.666992 1188 tablet_bootstrap.cc:654] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.667263 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]
I20250110 02:51:18.667729 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]
I20250110 02:51:18.668828 1184 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.669656 1184 ts_tablet_manager.cc:1397] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250110 02:51:18.670887 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c99830786e0e4b7b956e604dd3a5fcc0 [id=2af0614c37b44ebb9baa17b28b82f381]
I20250110 02:51:18.671108 1184 raft_consensus.cc:357] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.671923 1184 raft_consensus.cc:383] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.672139 1680 raft_consensus.cc:491] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.672286 1184 raft_consensus.cc:738] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.672289 1680 raft_consensus.cc:513] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.673609 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e183122d13a4c38988bf2fe6f7a3694" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.673866 1181 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.674069 1181 ts_tablet_manager.cc:1397] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20250110 02:51:18.674012 1184 consensus_queue.cc:260] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:18.674152 994 leader_election.cc:343] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.674547 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 [id=2fe05e5dfd244de7838c0642fa521be3]
I20250110 02:51:18.674990 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 [id=2fe86361000e43eda29c3a39e78beeee]
I20250110 02:51:18.675087 1184 ts_tablet_manager.cc:1428] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:18.675213 1181 raft_consensus.cc:357] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.675595 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]
I20250110 02:51:18.676060 1184 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:18.676651 996 leader_election.cc:343] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:18.676923 996 leader_election.cc:304] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.677245 1678 raft_consensus.cc:2743] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.676808 1680 leader_election.cc:290] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.677682 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]
I20250110 02:51:18.677757 1181 raft_consensus.cc:383] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.677935 1181 raft_consensus.cc:738] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.678303 1181 consensus_queue.cc:260] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.678484 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]
I20250110 02:51:18.678967 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]
I20250110 02:51:18.679179 1188 tablet_bootstrap.cc:492] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.679343 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]
I20250110 02:51:18.680330 1184 tablet_bootstrap.cc:654] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.680670 1188 ts_tablet_manager.cc:1397] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250110 02:51:18.681576 1188 raft_consensus.cc:357] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.682328 1181 ts_tablet_manager.cc:1428] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:18.686918 1181 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.690883 1181 tablet_bootstrap.cc:654] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.691448 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_627ff2f24b97485a9f948b72f8b39daf [id=3e8497573a584513ab1df4ba5ed27993]
I20250110 02:51:18.691870 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]
I20250110 02:51:18.692276 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_eb40720cee0c4ef08e2c5a49f1374c8e [id=40f4c299cde64d02bd2a420d11b6765f]
I20250110 02:51:18.692696 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e7baf08453f44f56a442f1d355c8a32c [id=41e43455ee7c45a6960721c33a18b4d8]
I20250110 02:51:18.693130 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_34ce8eea83364d1ca7d8cfa100115b45 [id=4696b770f7514e2f9d5aab8e16ad9eae]
I20250110 02:51:18.693567 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5e1c1dd369d94a688dabf41fda3795b6 [id=474dc245f50f45b4b43a1e6886a33c45]
I20250110 02:51:18.694600 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_70a31a6f02a443d68091578355b71adc [id=4946b19488484f39ad47b1118ce03359]
I20250110 02:51:18.695043 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_68d7c1c889464468adcd9153caef7ecc [id=497a56d44a114093934dbb9fde1b9f8a]
I20250110 02:51:18.695662 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3f51e7f8aaa7470fbeeb5459695efab1 [id=4b8aabec28ba49388402a7d16de036c5]
I20250110 02:51:18.696321 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_68a6c97dd17c47b28f987d6e8e778e37 [id=4d8aac2aeda44055ab88ed9f82e8efeb]
I20250110 02:51:18.696802 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a9587fbd6e924c49b1a9a1189bff5d63 [id=4e31223c509f46eba782e7c34f293c18]
I20250110 02:51:18.697631 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]
I20250110 02:51:18.698069 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]
I20250110 02:51:18.698534 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]
I20250110 02:51:18.699553 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_01c271ca90d2495b95e92eb25502cff2 [id=5580408664d94826a2f97912fb853728]
I20250110 02:51:18.700016 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]
I20250110 02:51:18.700529 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_93c93b5d73f24f75afbcb7660706db8a [id=5abe0f9c7bb9402c941464132d4ca4ae]
I20250110 02:51:18.701141 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_b064f366436b4b72bc8428d0bbb0c3df [id=5b2f66225983487abf161e772efb6ac2]
I20250110 02:51:18.701318 1188 raft_consensus.cc:383] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.701496 1188 raft_consensus.cc:738] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.701606 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_ce235042d71840c0a05ae04e253c9517 [id=5ba8eda7053b45e2b27844e97e5c513a]
I20250110 02:51:18.701835 1188 consensus_queue.cc:260] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.702648 1188 ts_tablet_manager.cc:1428] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250110 02:51:18.703341 1188 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.704391 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_375ea34c1b6d471ca632286504f259f0 [id=5d041eb06ad54278b7a327c69bbae96a]
I20250110 02:51:18.705638 1188 tablet_bootstrap.cc:654] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.707181 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_18eb7bac666944ddaf84da77edd4462b [id=5d3953c49f6f46b29e8ea2cd1f72f8d0]
I20250110 02:51:18.709455 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_befac4c75a404c7aa6d15f1fac47ce8b [id=614609c8b3154a4f85cfe592e4f2f182]
I20250110 02:51:18.710243 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d]
I20250110 02:51:18.711546 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]
I20250110 02:51:18.713727 1188 tablet_bootstrap.cc:492] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.713937 1188 ts_tablet_manager.cc:1397] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
W20250110 02:51:18.714324 994 consensus_peers.cc:487] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:18.715015 1184 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.715075 1181 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.715166 1184 ts_tablet_manager.cc:1397] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.004s
I20250110 02:51:18.715211 1181 ts_tablet_manager.cc:1397] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20250110 02:51:18.715730 1184 raft_consensus.cc:357] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.715879 1184 raft_consensus.cc:383] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.715981 1184 raft_consensus.cc:738] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.715881 1181 raft_consensus.cc:357] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.716084 1181 raft_consensus.cc:383] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.716183 1181 raft_consensus.cc:738] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.716197 1184 consensus_queue.cc:260] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.716375 1181 consensus_queue.cc:260] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.716791 1181 ts_tablet_manager.cc:1428] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:18.717048 1181 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.719219 1181 tablet_bootstrap.cc:654] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.720358 1184 ts_tablet_manager.cc:1428] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:18.720638 1184 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.721972 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]
I20250110 02:51:18.722409 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]
I20250110 02:51:18.722527 1768 raft_consensus.cc:491] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.722661 1768 raft_consensus.cc:513] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.722992 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]
I20250110 02:51:18.723239 1768 leader_election.cc:290] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.723297 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]
I20250110 02:51:18.723620 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.723693 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.723820 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_97ca49cb31634f579d9d84e0d4fc8057 [id=6acef2fac29441ceac28a2721295136a]
W20250110 02:51:18.724097 711 leader_election.cc:343] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.724239 709 leader_election.cc:343] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.724318 709 leader_election.cc:304] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.724488 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]
I20250110 02:51:18.724802 1768 raft_consensus.cc:2743] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.724851 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e [id=6eaaca051e6c496eaef0567ca4b21030]
I20250110 02:51:18.725242 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]
I20250110 02:51:18.725674 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_09cf33925be04c35b8a1304633c52217 [id=6f90cd9154c3419b8d3fd88395270251]
I20250110 02:51:18.726262 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]
I20250110 02:51:18.726619 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 [id=722b4aa4edcb4752a0df9a714e558255]
I20250110 02:51:18.726938 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_0de0a6680c0f4531a75432a618576660 [id=72c128c9bf8849638e64f4eb99ec4c7e]
I20250110 02:51:18.723130 1188 raft_consensus.cc:357] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.731053 1188 raft_consensus.cc:383] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.731133 1188 raft_consensus.cc:738] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.722544 1184 tablet_bootstrap.cc:654] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.732218 1188 consensus_queue.cc:260] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.732757 1188 ts_tablet_manager.cc:1428] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250110 02:51:18.733299 1188 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.735339 1188 tablet_bootstrap.cc:654] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.735813 1768 raft_consensus.cc:491] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.735941 1768 raft_consensus.cc:513] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.736128 1181 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.736248 1181 ts_tablet_manager.cc:1397] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250110 02:51:18.736668 1768 leader_election.cc:290] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.736908 1181 raft_consensus.cc:357] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.737077 1181 raft_consensus.cc:383] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.737185 1181 raft_consensus.cc:738] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.737301 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.737419 1181 consensus_queue.cc:260] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:18.737876 711 leader_election.cc:343] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.738765 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.740516 1181 ts_tablet_manager.cc:1428] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250110 02:51:18.741196 709 leader_election.cc:343] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.741334 709 leader_election.cc:304] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.743477 1768 raft_consensus.cc:2743] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.744580 1181 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.746063 1181 tablet_bootstrap.cc:654] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.747130 1726 raft_consensus.cc:491] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.747313 1726 raft_consensus.cc:513] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.747601 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]
I20250110 02:51:18.747841 1184 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.747938 1726 leader_election.cc:290] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.747988 1184 ts_tablet_manager.cc:1397] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.006s
I20250110 02:51:18.748059 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]
I20250110 02:51:18.748821 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]
I20250110 02:51:18.748768 1184 raft_consensus.cc:357] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:18.748859 853 leader_election.cc:343] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:18.748961 1184 raft_consensus.cc:383] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.749023 1184 raft_consensus.cc:738] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.749161 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6f9765add6c947e8b8154e36e6d358dc [id=7badf15e8d1543209edcffa451e3b48c]
I20250110 02:51:18.749290 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad0d561a8ef498ba13bc813d4cd82fa" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.749265 1184 consensus_queue.cc:260] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:18.749822 853 leader_election.cc:343] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.749951 853 leader_election.cc:304] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.750600 1726 raft_consensus.cc:2743] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.750914 1768 raft_consensus.cc:491] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.751071 1768 raft_consensus.cc:513] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.751760 1188 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.751767 1768 leader_election.cc:290] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.751884 1188 ts_tablet_manager.cc:1397] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20250110 02:51:18.752504 1188 raft_consensus.cc:357] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.752672 1188 raft_consensus.cc:383] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.752739 1188 raft_consensus.cc:738] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.752905 1188 consensus_queue.cc:260] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.753324 1188 ts_tablet_manager.cc:1428] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:18.753551 1188 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.755005 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5131fbd504e04f50b92a6a1fb6d0c055 [id=7c940859b1954178bfcb14fb882668f6]
I20250110 02:51:18.755091 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.755333 1188 tablet_bootstrap.cc:654] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.755415 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]
W20250110 02:51:18.755577 711 leader_election.cc:343] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.755842 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]
I20250110 02:51:18.756263 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]
I20250110 02:51:18.756554 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.756687 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80]
I20250110 02:51:18.757042 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_adb6100fd2ea489590831f0508f53e3e [id=811a01d9de1a4185a876251bd8d10512]
W20250110 02:51:18.757571 709 leader_election.cc:343] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.757721 709 leader_election.cc:304] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.758154 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_78653a7a5f0743f18b98b3e703c05a6c [id=8210fc764873462ca2238d58c2b61bfb]
I20250110 02:51:18.758914 1181 tablet_bootstrap.cc:492] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.759089 1181 ts_tablet_manager.cc:1397] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250110 02:51:18.759718 1181 raft_consensus.cc:357] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.759933 1181 raft_consensus.cc:383] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.759986 1181 raft_consensus.cc:738] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.760181 1181 consensus_queue.cc:260] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.763492 1181 ts_tablet_manager.cc:1428] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:18.765538 1181 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.767285 1181 tablet_bootstrap.cc:654] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.767282 1768 raft_consensus.cc:2743] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.769241 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c0c89321da6c41cbbf765c7562f60e95 [id=866547595c5d4cce9221971e917542eb]
I20250110 02:51:18.771889 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]
I20250110 02:51:18.772444 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]
I20250110 02:51:18.773314 1181 tablet_bootstrap.cc:492] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.773439 1181 ts_tablet_manager.cc:1397] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250110 02:51:18.774266 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]
I20250110 02:51:18.774171 1181 raft_consensus.cc:357] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.774410 1181 raft_consensus.cc:383] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.774521 1181 raft_consensus.cc:738] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.774803 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c7fa3a12842444fa9aa1758777787789 [id=88412101305b4d0a8dea71c08ff391aa]
I20250110 02:51:18.774752 1181 consensus_queue.cc:260] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.774931 1184 ts_tablet_manager.cc:1428] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.027s user 0.001s sys 0.018s
I20250110 02:51:18.778484 1181 ts_tablet_manager.cc:1428] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:18.778748 1181 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.778977 1188 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.779162 1188 ts_tablet_manager.cc:1397] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20250110 02:51:18.779367 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 [id=88d6d3dc5db2480aa3ec55acb2ec5cdc]
I20250110 02:51:18.779820 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]
I20250110 02:51:18.779865 1188 raft_consensus.cc:357] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.780079 1188 raft_consensus.cc:383] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.780164 1188 raft_consensus.cc:738] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.780485 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_60665b68321047d3a87e132a42f6c058 [id=8c72b8cb59bc4abdb98913a891ef023a]
I20250110 02:51:18.780928 1188 consensus_queue.cc:260] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.781543 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a993dbbdb6714184867df68d17523ca7 [id=8cf03d1b06f24347a2f8c4132f135d75]
I20250110 02:51:18.782364 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_62d79207c4534af2becc6ccc6da65f1e [id=8e8cecbb687a48009172ae19382c9b80]
I20250110 02:51:18.783962 1184 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.784571 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a [id=90471593be8d4c04b0ac2d50a96966e2]
I20250110 02:51:18.785161 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]
I20250110 02:51:18.785660 1184 tablet_bootstrap.cc:654] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.785662 1188 ts_tablet_manager.cc:1428] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:18.785898 1188 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.787792 1188 tablet_bootstrap.cc:654] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.792460 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_dd0bd0decd37446f891e12d29ab4955b [id=92d9235395b24e1eb75141de12dbe3e7]
I20250110 02:51:18.793119 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]
I20250110 02:51:18.793488 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 [id=9643d639878f4122bf42aeca200f9c17]
I20250110 02:51:18.793931 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5156f6b732b14eb6807a520bdfa15b62 [id=995906957594490ba9474338a3e342d2]
I20250110 02:51:18.794339 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09]
I20250110 02:51:18.794864 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_633f75bba62141e9a0075b23b2ec4bc0 [id=9b6434aff03e40a086c035ef1a731d3a]
I20250110 02:51:18.795737 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_26b5c54688724d26b6bf4f3587158e9c [id=9e880e06a6d648b7bc44b651084a14a3]
I20250110 02:51:18.796427 1184 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.796444 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 [id=a09ade04ca6745f3899557ae0b416eaa]
I20250110 02:51:18.796571 1184 ts_tablet_manager.cc:1397] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20250110 02:51:18.797616 1181 tablet_bootstrap.cc:654] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.797485 1184 raft_consensus.cc:357] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.797693 1184 raft_consensus.cc:383] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.797734 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]
I20250110 02:51:18.797784 1184 raft_consensus.cc:738] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.798034 1184 consensus_queue.cc:260] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.798657 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]
I20250110 02:51:18.800709 1726 raft_consensus.cc:491] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.800848 1726 raft_consensus.cc:513] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.801304 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c5d78661dad44d9f85ba6c91486185de [id=a6a89e8f33c74e61b95de2207ddbd585]
I20250110 02:51:18.801460 1726 leader_election.cc:290] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.801831 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_94cf6a20190647dd87d3616b069b0243 [id=a91e19e65fd74454b4b4d339a5cbe806]
I20250110 02:51:18.802855 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_bb1c63b5d7ee453a999514a777e71496 [id=ab05074705fa48438c82fd089d97be75]
I20250110 02:51:18.803287 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.803449 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3b60f335e3574a5bbf7ab20a93a64bed [id=b00159b02a3a4c2d877a7e8c7bf1fdeb]
I20250110 02:51:18.803930 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]
I20250110 02:51:18.803884 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.804111 774 raft_consensus.cc:2463] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:18.804281 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_742e66ce36c0463a978cb00de2f01135 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff]
W20250110 02:51:18.804972 853 leader_election.cc:343] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.805055 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]
I20250110 02:51:18.805104 1184 ts_tablet_manager.cc:1428] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250110 02:51:18.805364 853 leader_election.cc:304] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.805521 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]
I20250110 02:51:18.805764 1726 raft_consensus.cc:2798] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:18.805873 1726 raft_consensus.cc:491] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:18.805944 1726 raft_consensus.cc:3054] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.806208 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]
I20250110 02:51:18.806735 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c]
I20250110 02:51:18.807071 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534]
I20250110 02:51:18.807456 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]
I20250110 02:51:18.807806 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]
I20250110 02:51:18.808161 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_ca9f3a9eeaa845498cf76982cc9b59fd [id=bda31fe0278a4b9a95eedb85d71b2eff]
I20250110 02:51:18.808583 1726 raft_consensus.cc:513] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.808861 1188 tablet_bootstrap.cc:492] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.808982 1188 ts_tablet_manager.cc:1397] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250110 02:51:18.809034 1184 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.809649 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:18.809648 1188 raft_consensus.cc:357] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.809798 1188 raft_consensus.cc:383] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.809863 1188 raft_consensus.cc:738] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.810061 1188 consensus_queue.cc:260] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.811055 774 raft_consensus.cc:3054] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.812821 774 raft_consensus.cc:2463] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:18.813215 853 leader_election.cc:304] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:18.816144 1726 leader_election.cc:290] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.817649 1726 raft_consensus.cc:2798] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:18.817821 1726 raft_consensus.cc:695] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:18.818213 1726 consensus_queue.cc:237] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.819357 1184 tablet_bootstrap.cc:654] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.819512 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]
I20250110 02:51:18.820060 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]
I20250110 02:51:18.821396 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2009672da4c945ec8fbbac6f5fe0988a [id=c409e93003184d2282188bd417dd3580]
I20250110 02:51:18.827515 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]
I20250110 02:51:18.828500 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d]
I20250110 02:51:18.829366 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_09dd77d568b8468cbad2375415ecb197 [id=c6fdcf73aa2a471faeaca5a4a8f70258]
I20250110 02:51:18.829916 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 [id=c758d6f1cb1e4e48bee42c36a5df0e04]
I20250110 02:51:18.830504 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc]
I20250110 02:51:18.831053 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_48bb129a959843fcbad0b8d6344c6660 [id=c9d3b6064ff04c18a8d5e8f09a55a353]
I20250110 02:51:18.834738 1188 ts_tablet_manager.cc:1428] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.026s user 0.013s sys 0.013s
I20250110 02:51:18.834972 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_38cc13d405674486842b52efad1baed0 [id=cba096e296394b3ca68bc08f9770efcd]
I20250110 02:51:18.835143 1188 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.835412 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2e1a57f017f844aba03084fd604240cf [id=cdcb9c6db16b4d11ade1bd993b0756fc]
I20250110 02:51:18.836051 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf [id=cf302e964fd54aa7bb808f9801488d67]
I20250110 02:51:18.836521 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91]
I20250110 02:51:18.837298 1188 tablet_bootstrap.cc:654] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.837283 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]
I20250110 02:51:18.840171 1181 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.840341 1181 ts_tablet_manager.cc:1397] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.005s
I20250110 02:51:18.841149 1181 raft_consensus.cc:357] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.841449 1181 raft_consensus.cc:383] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.841540 1181 raft_consensus.cc:738] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.841800 1181 consensus_queue.cc:260] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.844224 1181 ts_tablet_manager.cc:1428] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:18.844493 1181 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.845336 1188 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.845506 1188 ts_tablet_manager.cc:1397] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20250110 02:51:18.846197 1188 raft_consensus.cc:357] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.846468 1188 raft_consensus.cc:383] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.846568 1188 raft_consensus.cc:738] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.846808 1188 consensus_queue.cc:260] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.846912 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8cf4c26686b84eb59c11d17010f8a679 [id=d37919aec6fc44ba9c1800b84d5fe13a]
I20250110 02:51:18.847445 1181 tablet_bootstrap.cc:654] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.847501 1188 ts_tablet_manager.cc:1428] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:18.847853 1188 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.848201 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]
I20250110 02:51:18.848676 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]
I20250110 02:51:18.849355 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e55025c253874d759eae8dd088e59196 [id=db8c2210559d44c6bd41a41f4d49ee21]
I20250110 02:51:18.849965 1188 tablet_bootstrap.cc:654] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.852089 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]
I20250110 02:51:18.854866 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8835f1dac47741f69e9a48aed136cde0 [id=e8a9e62e18bf419e807e689558db87ea]
I20250110 02:51:18.855046 1188 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:18.855232 1188 ts_tablet_manager.cc:1397] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20250110 02:51:18.855604 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]
I20250110 02:51:18.856061 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_35002d6ab8564fe1b93188d7ce973d3f [id=ed3a9f13f7794c3f9ee6917bf6ecff25]
I20250110 02:51:18.856091 1188 raft_consensus.cc:357] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.856328 1188 raft_consensus.cc:383] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.856455 1188 raft_consensus.cc:738] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.856528 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]
I20250110 02:51:18.856690 1188 consensus_queue.cc:260] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.857093 1184 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.857277 1184 ts_tablet_manager.cc:1397] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20250110 02:51:18.857285 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]
I20250110 02:51:18.857694 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]
I20250110 02:51:18.858413 1188 ts_tablet_manager.cc:1428] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:18.858834 1188 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:18.859107 1184 raft_consensus.cc:357] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.859403 1184 raft_consensus.cc:383] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.859413 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_77792a05daee44cfaaebbe693ef3cd65 [id=f1926d48a14c45699ed056c8b1818b4e]
I20250110 02:51:18.859494 1184 raft_consensus.cc:738] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.860076 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]
I20250110 02:51:18.859814 1184 consensus_queue.cc:260] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.860481 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_37fa37bb314c4af39a4bb95697714acb [id=f54f0ab6a3d140ae9ceac1818d68f7bb]
I20250110 02:51:18.860533 1184 ts_tablet_manager.cc:1428] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:18.860874 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_7cebf0b5d59d4f068fc8dce07001bd12 [id=f686f55e175d4a0bbdaa0d8a98d63791]
I20250110 02:51:18.861280 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]
I20250110 02:51:18.861759 1181 tablet_bootstrap.cc:492] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.861902 1181 ts_tablet_manager.cc:1397] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20250110 02:51:18.862541 1181 raft_consensus.cc:357] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.862730 1181 raft_consensus.cc:383] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.862848 1181 raft_consensus.cc:738] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.863044 1181 consensus_queue.cc:260] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.863165 1184 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:18.863476 1181 ts_tablet_manager.cc:1428] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:18.863828 1188 tablet_bootstrap.cc:654] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.865242 1184 tablet_bootstrap.cc:654] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.865864 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_e8d197a297e647b5ad3b516ff8054704 [id=f733132df99b481ea5a2117fe65a7fc4]
I20250110 02:51:18.868324 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_34eb9282d67443a2849ec525cdaa9e7b [id=f73acd74cc49421ea2e999c9de28b4ae]
I20250110 02:51:18.868916 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]
I20250110 02:51:18.870033 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]
I20250110 02:51:18.872077 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_c8df69267ed3452192228c2106e18a47 [id=fa0cfecca12c4ddcade02cc7a553edf1]
I20250110 02:51:18.874495 1680 raft_consensus.cc:491] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.874639 1680 raft_consensus.cc:513] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.875291 1680 leader_election.cc:290] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.876009 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.876523 994 leader_election.cc:343] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.877135 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:18.877337 774 raft_consensus.cc:2463] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:18.877967 996 leader_election.cc:304] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:18.878484 1680 raft_consensus.cc:2798] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:18.878598 1680 raft_consensus.cc:491] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:18.878696 1680 raft_consensus.cc:3054] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.880400 1181 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:18.881286 1680 raft_consensus.cc:513] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.881970 1181 tablet_bootstrap.cc:654] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.882324 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:18.883131 994 leader_election.cc:343] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.886445 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:18.886636 774 raft_consensus.cc:3054] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.889539 774 raft_consensus.cc:2463] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:18.890071 996 leader_election.cc:304] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:18.890990 1678 raft_consensus.cc:2798] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:18.891142 1678 raft_consensus.cc:695] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:18.891407 1678 consensus_queue.cc:237] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.892865 1680 leader_election.cc:290] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:18.894184 511 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8b2297377bc4e5095826e30523eede7 consensus state has no leader
I20250110 02:51:18.891410 1181 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.900228 1181 ts_tablet_manager.cc:1397] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20250110 02:51:18.902694 1181 raft_consensus.cc:357] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.902942 1181 raft_consensus.cc:383] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.903033 1181 raft_consensus.cc:738] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.903769 1181 consensus_queue.cc:260] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.905226 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 646 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd"
I20250110 02:51:18.905643 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d8d40556dc348e8aa8255d116251a10" candidate_term: 4 candidate_status { last_received { term: 3 index: 646 } } ignore_live_leader: false dest_uuid: "8bc934cc708045e9858ee8f8e64660cd" is_pre_election: true
W20250110 02:51:18.905885 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 92702.7us
I20250110 02:51:18.906446 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:18.907398 1181 ts_tablet_manager.cc:1428] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250110 02:51:18.907729 1181 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:18.908090 512 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 926228us
W20250110 02:51:18.912555 853 leader_election.cc:343] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.916523 442 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.61:44449 timed out after 0.561s (SENT) [suppressed 7 similar messages]
I20250110 02:51:18.928303 1680 raft_consensus.cc:491] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.928694 1825 raft_consensus.cc:491] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.928825 1680 raft_consensus.cc:513] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.928830 1825 raft_consensus.cc:513] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.929533 1825 leader_election.cc:290] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.930083 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:18.930428 1846 tablet.cc:1721] T bd6d4c4631994a5792200ba4d4c906f0 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:18.930557 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.931443 709 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.931782 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a4745356e3423f9d4892d242da1c48" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.932011 916 raft_consensus.cc:2463] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:18.932438 994 leader_election.cc:304] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:18.933125 1759 raft_consensus.cc:2798] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:18.933216 1759 raft_consensus.cc:491] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:18.933307 1759 raft_consensus.cc:3054] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:18.935830 1759 raft_consensus.cc:513] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.936089 1734 consensus_queue.cc:1035] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:18.937328 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a4745356e3423f9d4892d242da1c48" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:18.937505 916 raft_consensus.cc:3054] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:18.938305 711 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.938522 711 leader_election.cc:304] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.938987 1825 raft_consensus.cc:2743] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.940712 916 raft_consensus.cc:2463] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:18.941411 994 leader_election.cc:304] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:18.941769 1734 raft_consensus.cc:2798] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:18.941931 1734 raft_consensus.cc:695] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:18.942224 1734 consensus_queue.cc:237] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.947880 1680 leader_election.cc:290] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.948964 1759 leader_election.cc:290] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:18.949054 996 leader_election.cc:343] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 46a4745356e3423f9d4892d242da1c48
I20250110 02:51:18.950219 1825 raft_consensus.cc:491] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:18.950354 996 leader_election.cc:343] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 46a4745356e3423f9d4892d242da1c48
I20250110 02:51:18.950394 1825 raft_consensus.cc:513] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.951133 1825 leader_election.cc:290] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:18.952031 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:18.952210 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:18.952646 709 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.952876 711 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:18.953037 711 leader_election.cc:304] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:18.953542 1825 raft_consensus.cc:2743] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:18.954869 556 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:46668 (request call id 37) took 997 ms (client timeout 1000 ms). Trace:
W20250110 02:51:18.958734 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskadt4RW/build/asan/bin/kudu pid 494: Timed out: CheckLeaks RPC to 127.0.104.61:44449 timed out after 1.000s (SENT)
I20250110 02:51:18.958922 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskadt4RW/build/asan/bin/kudu with pid 494
I20250110 02:51:18.960218 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_a171d415e02a4113afd448d388d8f124 [id=fa45e9ce8e684b3191fb25c7b33246ed]
I20250110 02:51:18.962250 1181 tablet_bootstrap.cc:654] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:18.963359 1184 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:18.963522 1184 ts_tablet_manager.cc:1397] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.029s
I20250110 02:51:18.965124 1184 raft_consensus.cc:357] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.965380 1184 raft_consensus.cc:383] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.965471 1184 raft_consensus.cc:738] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.965973 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6a8868f4b22043c193efdd181759b4f4 [id=fd05c3a951c64cbf81dfd168728165f4]
I20250110 02:51:18.969821 1844 tablet.cc:1721] T bd6d4c4631994a5792200ba4d4c906f0 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:18.980027 1770 raft_consensus.cc:491] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:18.980311 1770 raft_consensus.cc:513] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.983261 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]
I20250110 02:51:18.983841 1804 catalog_manager.cc:670] Loaded metadata for table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2]
I20250110 02:51:18.988270 1184 consensus_queue.cc:260] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:18.994174 1770 leader_election.cc:290] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:18.994575 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf711522866d46439d7ee69aa63d7f39" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:18.995244 994 leader_election.cc:343] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:18.995977 996 leader_election.cc:343] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:18.996099 996 leader_election.cc:304] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:18.996701 1770 raft_consensus.cc:2743] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:18.997763 1181 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:18.997946 1181 ts_tablet_manager.cc:1397] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.090s user 0.023s sys 0.007s
I20250110 02:51:18.998800 1181 raft_consensus.cc:357] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:18.999027 1181 raft_consensus.cc:383] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:18.999188 1181 raft_consensus.cc:738] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:18.999464 1181 consensus_queue.cc:260] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.000087 1181 ts_tablet_manager.cc:1428] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.002584 1181 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.004307 1825 raft_consensus.cc:491] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.004441 1825 raft_consensus.cc:513] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.005062 1825 leader_election.cc:290] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.005489 1181 tablet_bootstrap.cc:654] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.005563 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.005645 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.006270 711 leader_election.cc:343] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.006467 709 leader_election.cc:343] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.006665 709 leader_election.cc:304] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.007063 1825 raft_consensus.cc:2743] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.010360 1832 raft_consensus.cc:491] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.010551 1832 raft_consensus.cc:513] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.011336 1832 leader_election.cc:290] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:19.012763 853 leader_election.cc:343] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:19.013433 1849 raft_consensus.cc:491] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.013593 1849 raft_consensus.cc:513] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.013972 853 leader_election.cc:343] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:19.014120 853 leader_election.cc:304] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.014505 1849 leader_election.cc:290] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.015192 1832 raft_consensus.cc:2743] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.020270 1184 ts_tablet_manager.cc:1428] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.057s user 0.001s sys 0.002s
I20250110 02:51:19.023937 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.024639 994 leader_election.cc:343] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.020610 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.026973 775 raft_consensus.cc:2463] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.027130 1184 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.027552 996 leader_election.cc:304] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:19.027949 1849 raft_consensus.cc:2798] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.028111 1849 raft_consensus.cc:491] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.028215 1849 raft_consensus.cc:3054] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.029985 1184 tablet_bootstrap.cc:654] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.033243 1849 raft_consensus.cc:513] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.033422 1188 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.033572 1188 ts_tablet_manager.cc:1397] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.175s user 0.066s sys 0.101s
I20250110 02:51:19.034399 1188 raft_consensus.cc:357] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.034643 1188 raft_consensus.cc:383] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.034737 1188 raft_consensus.cc:738] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.034674 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.034955 1188 consensus_queue.cc:260] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.035377 994 leader_election.cc:343] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.035483 1188 ts_tablet_manager.cc:1428] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:19.035790 1188 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.037833 1188 tablet_bootstrap.cc:654] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.042223 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:19.042657 775 raft_consensus.cc:3054] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.043646 1832 raft_consensus.cc:491] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.043766 1832 raft_consensus.cc:513] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.044332 1832 leader_election.cc:290] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.046344 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14f06ba7d0f44418693e36f6bc6d888" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.046636 1061 raft_consensus.cc:2463] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.047324 853 leader_election.cc:304] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
W20250110 02:51:19.047648 853 leader_election.cc:343] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888
I20250110 02:51:19.047660 1849 leader_election.cc:290] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.048157 1825 raft_consensus.cc:491] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.048245 1832 raft_consensus.cc:2798] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.048384 1832 raft_consensus.cc:491] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.048475 1832 raft_consensus.cc:3054] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.048326 1825 raft_consensus.cc:513] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.049095 1825 leader_election.cc:290] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.049623 775 raft_consensus.cc:2463] T 8bc89802926543738cb04c75821ab693 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.050767 1849 raft_consensus.cc:491] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.050993 1188 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.050935 1849 raft_consensus.cc:513] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.051103 1188 ts_tablet_manager.cc:1397] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20250110 02:51:19.051661 1849 leader_election.cc:290] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.051748 1832 raft_consensus.cc:513] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.052434 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.052623 775 raft_consensus.cc:2463] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.052964 996 leader_election.cc:304] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:19.053265 1849 raft_consensus.cc:2798] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.053366 996 leader_election.cc:304] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:19.053483 1849 raft_consensus.cc:695] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.053766 1678 raft_consensus.cc:2798] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.053862 1678 raft_consensus.cc:491] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.053951 1678 raft_consensus.cc:3054] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.053810 1849 consensus_queue.cc:237] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.055173 1770 raft_consensus.cc:491] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.055328 1770 raft_consensus.cc:513] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.056010 1770 leader_election.cc:290] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.055999 1678 raft_consensus.cc:513] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.056504 1678 leader_election.cc:290] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.056936 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.057658 711 leader_election.cc:343] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.057884 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:19.058084 774 raft_consensus.cc:3054] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.060238 774 raft_consensus.cc:2463] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.061012 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.061846 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14f06ba7d0f44418693e36f6bc6d888" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
W20250110 02:51:19.061902 709 leader_election.cc:343] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.062038 1061 raft_consensus.cc:3054] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.062011 709 leader_election.cc:304] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.062237 996 leader_election.cc:304] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:19.062433 853 leader_election.cc:343] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888
W20250110 02:51:19.062597 996 leader_election.cc:343] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:19.062795 1825 raft_consensus.cc:2743] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.063721 1849 raft_consensus.cc:2798] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.063890 1849 raft_consensus.cc:695] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.064098 1061 raft_consensus.cc:2463] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.064227 1849 consensus_queue.cc:237] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.051877 1188 raft_consensus.cc:357] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.052476 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.064865 1188 raft_consensus.cc:383] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.064980 1188 raft_consensus.cc:738] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.057117 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a9eaf548274b73a5652ba314dab70c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.057600 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e744cdc1fcb94d46b48b684763ecf1bf" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.065217 1188 consensus_queue.cc:260] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.065460 917 raft_consensus.cc:2463] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.065763 853 leader_election.cc:304] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.065896 1188 ts_tablet_manager.cc:1428] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250110 02:51:19.065999 1831 raft_consensus.cc:2798] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.066146 1831 raft_consensus.cc:695] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.066220 1188 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.066413 1831 consensus_queue.cc:237] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.067476 1832 leader_election.cc:290] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:19.067286 994 leader_election.cc:343] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.071276 994 leader_election.cc:343] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.071645 994 leader_election.cc:304] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.072053 1678 raft_consensus.cc:2798] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.072152 1184 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.072147 1678 raft_consensus.cc:491] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.072218 1678 raft_consensus.cc:3054] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.072288 1184 ts_tablet_manager.cc:1397] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.006s
I20250110 02:51:19.072453 1188 tablet_bootstrap.cc:654] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.073048 1184 raft_consensus.cc:357] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.073227 1184 raft_consensus.cc:383] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.073287 1184 raft_consensus.cc:738] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.073511 1184 consensus_queue.cc:260] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.074031 1184 ts_tablet_manager.cc:1428] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:19.076159 1184 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.080853 1184 tablet_bootstrap.cc:654] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.085206 1678 raft_consensus.cc:513] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.086477 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e744cdc1fcb94d46b48b684763ecf1bf" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.086737 917 raft_consensus.cc:3054] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.087891 1181 tablet_bootstrap.cc:492] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.088037 1181 ts_tablet_manager.cc:1397] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.085s user 0.030s sys 0.036s
I20250110 02:51:19.088766 1181 raft_consensus.cc:357] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.088944 1181 raft_consensus.cc:383] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.088994 1181 raft_consensus.cc:738] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.089176 1181 consensus_queue.cc:260] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.089435 1770 raft_consensus.cc:491] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.089598 917 raft_consensus.cc:2463] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.089680 1181 ts_tablet_manager.cc:1428] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:19.089584 1770 raft_consensus.cc:513] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.089972 1181 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.090188 1770 leader_election.cc:290] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.090809 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.090874 994 leader_election.cc:304] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.091024 775 raft_consensus.cc:2463] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.091176 1770 raft_consensus.cc:2798] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.091300 1770 raft_consensus.cc:695] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.091409 996 leader_election.cc:304] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:19.091569 1770 consensus_queue.cc:237] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.092538 1759 raft_consensus.cc:2798] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.092640 1759 raft_consensus.cc:491] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.092725 1759 raft_consensus.cc:3054] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.092825 1181 tablet_bootstrap.cc:654] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.093745 1804 tablet_loader.cc:96] loaded metadata for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_38cc13d405674486842b52efad1baed0 [id=cba096e296394b3ca68bc08f9770efcd])
I20250110 02:51:19.094273 1804 tablet_loader.cc:96] loaded metadata for tablet 00618f611d4c4a84a988125c5d175efc (table default.Table_6c7eac9722834633b607ad9b97e5947d [id=fda1f290a7fb4b46a24125ae643d52b2])
I20250110 02:51:19.095778 1759 raft_consensus.cc:513] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.096405 1759 leader_election.cc:290] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.096594 1825 raft_consensus.cc:491] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.096789 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:19.096738 1825 raft_consensus.cc:513] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.096977 775 raft_consensus.cc:3054] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.097384 1825 leader_election.cc:290] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.098620 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:19.099200 994 leader_election.cc:343] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.098685 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.099570 775 raft_consensus.cc:2463] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:19.099838 994 leader_election.cc:343] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.100375 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.100834 996 leader_election.cc:304] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:19.101320 1680 raft_consensus.cc:2798] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.101490 1680 raft_consensus.cc:695] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.101794 1680 consensus_queue.cc:237] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.102172 1678 leader_election.cc:290] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:19.103212 711 leader_election.cc:343] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.105322 996 leader_election.cc:343] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:19.106076 1825 raft_consensus.cc:491] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.106230 1825 raft_consensus.cc:513] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.106871 1825 leader_election.cc:290] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.108044 465 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Refusing update from remote peer 5d8d40556dc348e8aa8255d116251a10: Log matching property violated. Preceding OpId in replica: term: 3 index: 646. Preceding OpId from leader: term: 4 index: 647. (index mismatch)
I20250110 02:51:19.108016 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.108510 1804 tablet_loader.cc:96] loaded metadata for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5])
I20250110 02:51:19.108760 1798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 647, Last known committed idx: 646, Time since last communication: 0.000s
W20250110 02:51:19.109601 711 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.110395 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.110414 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.111238 709 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.111341 709 leader_election.cc:304] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.111480 1181 tablet_bootstrap.cc:492] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.111586 1181 ts_tablet_manager.cc:1397] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
W20250110 02:51:19.111846 709 leader_election.cc:343] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.111935 709 leader_election.cc:304] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.112169 1768 raft_consensus.cc:2743] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.112327 1768 raft_consensus.cc:2743] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.112313 1181 raft_consensus.cc:357] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.112506 1181 raft_consensus.cc:383] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.112659 1181 raft_consensus.cc:738] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.112874 1181 consensus_queue.cc:260] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.113915 1181 ts_tablet_manager.cc:1428] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:19.114199 1181 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.115573 1804 tablet_loader.cc:96] loaded metadata for tablet 023b333c321a40dcb83a67ec5fb8ff0e (table default.Table_5e1c1dd369d94a688dabf41fda3795b6 [id=474dc245f50f45b4b43a1e6886a33c45])
I20250110 02:51:19.116176 1181 tablet_bootstrap.cc:654] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.117534 1804 tablet_loader.cc:96] loaded metadata for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77])
I20250110 02:51:19.119103 1804 tablet_loader.cc:96] loaded metadata for tablet 0558a2c03f6b42a6befdc48e3af23a6d (table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e])
I20250110 02:51:19.120606 1804 tablet_loader.cc:96] loaded metadata for tablet 069d19975e2a4bb0a23b4e9983f411d3 (table default.Table_befac4c75a404c7aa6d15f1fac47ce8b [id=614609c8b3154a4f85cfe592e4f2f182])
I20250110 02:51:19.122247 1804 tablet_loader.cc:96] loaded metadata for tablet 071bbce1706c43b6bb690c7de1ead3fe (table default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85])
I20250110 02:51:19.123289 1804 tablet_loader.cc:96] loaded metadata for tablet 07acc4681ae94a069003ed75fcce5141 (table default.Table_5156f6b732b14eb6807a520bdfa15b62 [id=995906957594490ba9474338a3e342d2])
I20250110 02:51:19.123965 1680 raft_consensus.cc:491] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.124115 1680 raft_consensus.cc:513] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.124567 1804 tablet_loader.cc:96] loaded metadata for tablet 07c4c653a5954d2789abd1611e3c64d0 (table default.Table_34eb9282d67443a2849ec525cdaa9e7b [id=f73acd74cc49421ea2e999c9de28b4ae])
I20250110 02:51:19.124655 1680 leader_election.cc:290] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.125177 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585320c0970c4097b195caaae33789b5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.125398 917 raft_consensus.cc:2463] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.125725 1862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8d40556dc348e8aa8255d116251a10. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:19.126204 1862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:19.127272 994 leader_election.cc:304] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.127691 1680 raft_consensus.cc:2798] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.127813 1680 raft_consensus.cc:491] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.127898 1680 raft_consensus.cc:3054] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.129777 1825 raft_consensus.cc:491] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.129916 1825 raft_consensus.cc:513] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.130475 1804 tablet_loader.cc:96] loaded metadata for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_e55025c253874d759eae8dd088e59196 [id=db8c2210559d44c6bd41a41f4d49ee21])
I20250110 02:51:19.130699 1825 leader_election.cc:290] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.130841 1680 raft_consensus.cc:513] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.131059 1804 tablet_loader.cc:96] loaded metadata for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06])
I20250110 02:51:19.131356 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.131940 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585320c0970c4097b195caaae33789b5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.132180 917 raft_consensus.cc:3054] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:19.132328 711 leader_election.cc:343] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.132995 996 leader_election.cc:343] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 585320c0970c4097b195caaae33789b5
W20250110 02:51:19.133601 996 leader_election.cc:343] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 585320c0970c4097b195caaae33789b5
I20250110 02:51:19.134585 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.135146 1804 tablet_loader.cc:96] loaded metadata for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215])
W20250110 02:51:19.135171 709 leader_election.cc:343] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.135315 709 leader_election.cc:304] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.135775 1825 raft_consensus.cc:2743] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.137075 1804 tablet_loader.cc:96] loaded metadata for tablet 1253c54d63cc47f081e676851305bc4b (table default.Table_8a8d43a5e45c47f083ed451f5b30bf6b [id=2387784c829c4e8bb0ecc7d63bb42111])
I20250110 02:51:19.137418 917 raft_consensus.cc:2463] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.137866 994 leader_election.cc:304] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.138177 1770 raft_consensus.cc:2798] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.138219 1680 leader_election.cc:290] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.138341 1770 raft_consensus.cc:695] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.138638 1770 consensus_queue.cc:237] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.140272 1804 tablet_loader.cc:96] loaded metadata for tablet 13683fbe9d654575b28108de8b77f943 (table default.Table_a73b4a8bd4a741d68bae45ba1bd417e4 [id=2fe05e5dfd244de7838c0642fa521be3])
I20250110 02:51:19.141180 1188 tablet_bootstrap.cc:492] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.141386 1188 ts_tablet_manager.cc:1397] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.075s user 0.021s sys 0.000s
I20250110 02:51:19.142278 1188 raft_consensus.cc:357] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.142645 1188 raft_consensus.cc:383] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.142692 1184 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.142760 1188 raft_consensus.cc:738] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.142988 1184 ts_tablet_manager.cc:1397] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.008s
I20250110 02:51:19.143030 1188 consensus_queue.cc:260] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.143792 1188 ts_tablet_manager.cc:1428] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.143815 1184 raft_consensus.cc:357] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.144080 1184 raft_consensus.cc:383] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.144196 1184 raft_consensus.cc:738] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.144436 1184 consensus_queue.cc:260] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.145601 1804 tablet_loader.cc:96] loaded metadata for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a (table default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80])
I20250110 02:51:19.146296 1804 tablet_loader.cc:96] loaded metadata for tablet 1cc0532c94eb401e9d299e79556b4cf1 (table default.Table_78653a7a5f0743f18b98b3e703c05a6c [id=8210fc764873462ca2238d58c2b61bfb])
I20250110 02:51:19.146751 1181 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.146733 1804 tablet_loader.cc:96] loaded metadata for tablet 1d27f9cb38434646822b6ba1fac53282 (table default.Table_01c271ca90d2495b95e92eb25502cff2 [id=5580408664d94826a2f97912fb853728])
I20250110 02:51:19.146890 1181 ts_tablet_manager.cc:1397] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.003s
I20250110 02:51:19.147128 1804 tablet_loader.cc:96] loaded metadata for tablet 1d2f4751d0f64e39b975529c6ac28d8a (table default.Table_28544fb9a8fd4edfbeb5cd13e63f2dcf [id=cf302e964fd54aa7bb808f9801488d67])
I20250110 02:51:19.147557 1804 tablet_loader.cc:96] loaded metadata for tablet 1d634dd0c2044f91b304fe3737330d88 (table default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86])
I20250110 02:51:19.147578 1181 raft_consensus.cc:357] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.147799 1181 raft_consensus.cc:383] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.147862 1181 raft_consensus.cc:738] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.147972 1804 tablet_loader.cc:96] loaded metadata for tablet 1df15c10ef2d4bf99af06a860c6f250f (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a])
I20250110 02:51:19.148465 1825 raft_consensus.cc:491] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.148590 1804 tablet_loader.cc:96] loaded metadata for tablet 1eaf7bbc981744f2bd78b97a3624448d (table default.Table_7fdb03687ec14267bfee2826ddb5a13b [id=14b73a8ae2744e1ea505735074b5c3ee])
I20250110 02:51:19.148605 1825 raft_consensus.cc:513] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.148947 1804 tablet_loader.cc:96] loaded metadata for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334])
I20250110 02:51:19.149286 1825 leader_election.cc:290] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.149643 1188 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.150072 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.150934 711 leader_election.cc:343] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.150955 1181 consensus_queue.cc:260] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.151717 1188 tablet_bootstrap.cc:654] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.152118 1181 ts_tablet_manager.cc:1428] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:19.152530 1181 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.153478 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.153949 709 leader_election.cc:343] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.154045 709 leader_election.cc:304] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.154289 1181 tablet_bootstrap.cc:654] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.155328 1825 raft_consensus.cc:2743] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.155961 1184 ts_tablet_manager.cc:1428] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20250110 02:51:19.156247 1184 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.158059 1184 tablet_bootstrap.cc:654] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.159227 1831 raft_consensus.cc:491] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.159373 1831 raft_consensus.cc:513] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.159932 1831 leader_election.cc:290] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.160706 1804 tablet_loader.cc:96] loaded metadata for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd])
I20250110 02:51:19.160895 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.161406 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b91f351e02d406f8675eb44c4c04ced" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.161576 1188 tablet_bootstrap.cc:492] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.161680 775 raft_consensus.cc:2388] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:19.161697 1188 ts_tablet_manager.cc:1397] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250110 02:51:19.162477 1188 raft_consensus.cc:357] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.162658 1188 raft_consensus.cc:383] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.162743 1188 raft_consensus.cc:738] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.162793 853 leader_election.cc:304] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.162951 1188 consensus_queue.cc:260] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.163177 1831 raft_consensus.cc:3054] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.163542 1188 ts_tablet_manager.cc:1428] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.163810 1188 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.165536 1188 tablet_bootstrap.cc:654] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.166628 1862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:19.166857 1862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:19.167081 1804 tablet_loader.cc:96] loaded metadata for tablet 248055ca83de4e62a6b7e908c53d3d9a (table default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88])
I20250110 02:51:19.167078 1831 raft_consensus.cc:2743] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.167752 1804 tablet_loader.cc:96] loaded metadata for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb])
I20250110 02:51:19.169569 1804 tablet_loader.cc:96] loaded metadata for tablet 2abc408e62034459815486648565e7ce (table default.Table_37fa37bb314c4af39a4bb95697714acb [id=f54f0ab6a3d140ae9ceac1818d68f7bb])
I20250110 02:51:19.170172 1804 tablet_loader.cc:96] loaded metadata for tablet 2b2b5e3c14244418912f344b7519995d (table default.Table_48bb129a959843fcbad0b8d6344c6660 [id=c9d3b6064ff04c18a8d5e8f09a55a353])
I20250110 02:51:19.170540 1804 tablet_loader.cc:96] loaded metadata for tablet 2da1ee1add01482cbf651bd2b4753dc1 (table default.Table_f1a7c1c5aa2b4c29bb620b0b2e3e8df6 [id=722b4aa4edcb4752a0df9a714e558255])
I20250110 02:51:19.175770 1181 tablet_bootstrap.cc:492] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.176012 1181 ts_tablet_manager.cc:1397] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.001s
I20250110 02:51:19.175952 1770 consensus_queue.cc:1035] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.176443 1184 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.176607 1184 ts_tablet_manager.cc:1397] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20250110 02:51:19.177521 1804 tablet_loader.cc:96] loaded metadata for tablet 2f0f754966c04074a568c05063591990 (table default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9])
I20250110 02:51:19.178141 1181 raft_consensus.cc:357] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.178485 1181 raft_consensus.cc:383] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.179032 1181 raft_consensus.cc:738] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.179312 1181 consensus_queue.cc:260] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.180150 1188 tablet_bootstrap.cc:492] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.180306 1188 ts_tablet_manager.cc:1397] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250110 02:51:19.181017 1188 raft_consensus.cc:357] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.181236 1188 raft_consensus.cc:383] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.181315 1188 raft_consensus.cc:738] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.181535 1188 consensus_queue.cc:260] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.182171 1188 ts_tablet_manager.cc:1428] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:19.182518 1188 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:19.182925 994 consensus_peers.cc:487] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:19.183802 1184 raft_consensus.cc:357] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.184077 1184 raft_consensus.cc:383] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.184121 1181 ts_tablet_manager.cc:1428] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:19.184151 1188 tablet_bootstrap.cc:654] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.184223 1184 raft_consensus.cc:738] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.184445 1181 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.184533 1184 consensus_queue.cc:260] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.185298 1184 ts_tablet_manager.cc:1428] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:19.185771 1184 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.186125 1181 tablet_bootstrap.cc:654] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.187645 1184 tablet_bootstrap.cc:654] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.188602 1804 tablet_loader.cc:96] loaded metadata for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee])
I20250110 02:51:19.190703 1825 raft_consensus.cc:491] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.190884 1825 raft_consensus.cc:513] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.191638 1825 leader_election.cc:290] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.193075 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.193348 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:19.193626 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:19.193598 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.198473 1804 tablet_loader.cc:96] loaded metadata for tablet 33f9b3e74ee2431a85a22fefaacfa40e (table default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92])
I20250110 02:51:19.198978 1804 tablet_loader.cc:96] loaded metadata for tablet 340882f7684a4e7c9a95800117fc758b (table default.Table_2e1a57f017f844aba03084fd604240cf [id=cdcb9c6db16b4d11ade1bd993b0756fc])
I20250110 02:51:19.199775 1804 tablet_loader.cc:96] loaded metadata for tablet 3510e86314144a2ca012aacf3d78eacc (table default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0])
I20250110 02:51:19.199818 1188 tablet_bootstrap.cc:492] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.199930 1188 ts_tablet_manager.cc:1397] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
W20250110 02:51:19.200129 709 leader_election.cc:343] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.200326 1804 tablet_loader.cc:96] loaded metadata for tablet 35ec1dba54a247d5ba9f9e053aa3b645 (table default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e])
I20250110 02:51:19.200770 1804 tablet_loader.cc:96] loaded metadata for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3])
I20250110 02:51:19.200765 1188 raft_consensus.cc:357] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.200947 1188 raft_consensus.cc:383] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.201043 1188 raft_consensus.cc:738] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.201184 1804 tablet_loader.cc:96] loaded metadata for tablet 36c0aaf2050f481daa8cdac4761fe705 (table default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc])
I20250110 02:51:19.201278 1188 consensus_queue.cc:260] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.201907 1188 ts_tablet_manager.cc:1428] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:19.201929 1804 tablet_loader.cc:96] loaded metadata for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26])
I20250110 02:51:19.202302 1804 tablet_loader.cc:96] loaded metadata for tablet 397f3261b76a4c82a3c265e68662841f (table default.Table_62b76b94b4124bbebf4f5cd9ad7a6ba7 [id=88d6d3dc5db2480aa3ec55acb2ec5cdc])
I20250110 02:51:19.202569 1188 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.202687 1804 tablet_loader.cc:96] loaded metadata for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c])
W20250110 02:51:19.203889 711 leader_election.cc:343] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.204031 711 leader_election.cc:304] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.204572 1188 tablet_bootstrap.cc:654] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.204561 1825 raft_consensus.cc:2743] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.205559 995 connection.cc:537] server connection from 127.0.0.1:55354 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 02:51:19.206075 1804 tablet_loader.cc:96] loaded metadata for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb])
I20250110 02:51:19.206523 1804 tablet_loader.cc:96] loaded metadata for tablet 3ca895eca14748c19fdb7d363499b544 (table default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47])
I20250110 02:51:19.208613 1825 raft_consensus.cc:491] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.208803 1825 raft_consensus.cc:513] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:19.209235 712 connection.cc:537] server connection from 127.0.0.1:55720 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 02:51:19.209520 1825 leader_election.cc:290] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:19.209712 1124 connection.cc:537] client connection to 127.0.104.61:44449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 02:51:19.209882 1434 connection.cc:537] client connection to 127.0.104.61:44449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 02:51:19.210170 1434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 02:51:19.211122 1124 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:44449)
W20250110 02:51:19.211716 1124 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:44449)
W20250110 02:51:19.212126 1124 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:44449)
W20250110 02:51:19.216842 1124 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:44449)
I20250110 02:51:19.217140 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.218837 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.219583 709 leader_election.cc:343] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.220351 711 leader_election.cc:343] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.220494 711 leader_election.cc:304] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.221164 1825 raft_consensus.cc:2743] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.230304 917 raft_consensus.cc:3054] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:19.241073 854 connection.cc:537] client connection to 127.0.104.61:44449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 02:51:19.259061 1770 raft_consensus.cc:491] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.259248 1770 raft_consensus.cc:513] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.259855 1770 leader_election.cc:290] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.260222 1885 tablet.cc:1721] T bd6d4c4631994a5792200ba4d4c906f0 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250110 02:51:19.261523 996 leader_election.cc:343] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:19.262686 1184 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.262835 1804 tablet_loader.cc:96] loaded metadata for tablet 3d4f88a7002f413fbd63491fe8e1fe6d (table default.Table_2009672da4c945ec8fbbac6f5fe0988a [id=c409e93003184d2282188bd417dd3580])
I20250110 02:51:19.262910 1184 ts_tablet_manager.cc:1397] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.001s
I20250110 02:51:19.263474 1804 tablet_loader.cc:96] loaded metadata for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde])
I20250110 02:51:19.263640 1184 raft_consensus.cc:357] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.263885 1184 raft_consensus.cc:383] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.263983 1184 raft_consensus.cc:738] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.264202 1184 consensus_queue.cc:260] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.264735 1184 ts_tablet_manager.cc:1428] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:19.267103 1184 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.268929 1831 raft_consensus.cc:491] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.269191 1181 tablet_bootstrap.cc:492] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.269078 1831 raft_consensus.cc:513] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.269330 1181 ts_tablet_manager.cc:1397] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.085s user 0.014s sys 0.034s
I20250110 02:51:19.269753 1831 leader_election.cc:290] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.270092 1181 raft_consensus.cc:357] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.270578 1181 raft_consensus.cc:383] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.270684 1181 raft_consensus.cc:738] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.270854 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.270977 1181 consensus_queue.cc:260] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.271929 1184 tablet_bootstrap.cc:654] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.272660 442 connection.cc:537] client connection to 127.0.104.61:44449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 02:51:19.272830 442 leader_election.cc:336] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 02:51:19.272995 442 leader_election.cc:304] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fca9a92f47145fcaaf4f65038d6f558; no voters: 5d8d40556dc348e8aa8255d116251a10, 8bc934cc708045e9858ee8f8e64660cd
I20250110 02:51:19.273329 1862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 02:51:19.274909 1181 ts_tablet_manager.cc:1428] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:19.275264 1181 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:19.276451 1887 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:19.243317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:44449 (local address 127.0.104.3:42453)
0110 02:51:19.253969 (+ 10652us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:19.254000 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0110 02:51:19.254136 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 02:51:19.276014 (+ 21878us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":10515,"thread_start_us":10899,"threads_started":1}
I20250110 02:51:19.277146 1181 tablet_bootstrap.cc:654] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.280820 1884 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:19.241585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:44449 (local address 127.0.0.1:35140)
0110 02:51:19.254711 (+ 13126us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:19.254726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0110 02:51:19.254835 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 02:51:19.280387 (+ 25552us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":13018,"thread_start_us":12923,"threads_started":1}
W20250110 02:51:19.281033 1434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 -> Peer 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449): Couldn't send request to peer 8bc934cc708045e9858ee8f8e64660cd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:19.283377 1188 tablet_bootstrap.cc:492] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.283547 1188 ts_tablet_manager.cc:1397] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.081s user 0.026s sys 0.042s
I20250110 02:51:19.283614 1770 raft_consensus.cc:491] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.283735 1770 raft_consensus.cc:513] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.284302 1770 leader_election.cc:290] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.284390 1188 raft_consensus.cc:357] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.284621 1188 raft_consensus.cc:383] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.284689 1188 raft_consensus.cc:738] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.284986 1188 consensus_queue.cc:260] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.285208 996 leader_election.cc:343] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:19.285553 1188 ts_tablet_manager.cc:1428] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:19.285883 1188 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.287645 1188 tablet_bootstrap.cc:654] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.289825 1181 tablet_bootstrap.cc:492] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.290012 1181 ts_tablet_manager.cc:1397] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250110 02:51:19.290989 1181 raft_consensus.cc:357] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.291234 1181 raft_consensus.cc:383] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.291340 1181 raft_consensus.cc:738] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.291532 1181 consensus_queue.cc:260] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.297329 1181 ts_tablet_manager.cc:1428] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250110 02:51:19.297957 1181 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.299774 1770 raft_consensus.cc:491] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.299940 1181 tablet_bootstrap.cc:654] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.299991 1770 raft_consensus.cc:513] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.300773 1770 leader_election.cc:290] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:19.301831 996 leader_election.cc:343] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:19.303998 1804 tablet_loader.cc:96] loaded metadata for tablet 3fb805b3e537466ab7fe0185421ad5aa (table default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539])
I20250110 02:51:19.305507 1181 tablet_bootstrap.cc:492] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.305722 1181 ts_tablet_manager.cc:1397] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250110 02:51:19.305713 1804 tablet_loader.cc:96] loaded metadata for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09])
I20250110 02:51:19.306455 1181 raft_consensus.cc:357] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.306704 1181 raft_consensus.cc:383] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.306793 1181 raft_consensus.cc:738] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.307050 1181 consensus_queue.cc:260] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.307605 1181 ts_tablet_manager.cc:1428] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:19.308487 1184 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.308640 1184 ts_tablet_manager.cc:1397] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.002s
I20250110 02:51:19.309441 1184 raft_consensus.cc:357] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.309675 1184 raft_consensus.cc:383] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.309754 1184 raft_consensus.cc:738] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.310027 1831 raft_consensus.cc:491] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.310210 1831 raft_consensus.cc:513] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.311059 1831 leader_election.cc:290] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.311650 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef72e05f8ed4d46b759c7a04497653b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.311856 1061 raft_consensus.cc:2463] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.312224 853 leader_election.cc:304] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.312544 1181 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.312592 1831 raft_consensus.cc:2798] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.312672 1831 raft_consensus.cc:491] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.312793 1831 raft_consensus.cc:3054] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.314549 1181 tablet_bootstrap.cc:654] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.315312 1831 raft_consensus.cc:513] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.316210 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef72e05f8ed4d46b759c7a04497653b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.316392 1061 raft_consensus.cc:3054] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.319340 1061 raft_consensus.cc:2463] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.319809 853 leader_election.cc:304] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.320122 1832 raft_consensus.cc:2798] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.320236 1832 raft_consensus.cc:695] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.320518 1832 consensus_queue.cc:237] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.321748 1886 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:19.242228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:44449 (local address 127.0.104.2:44897)
0110 02:51:19.321114 (+ 78886us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:19.321221 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":78764,"threads_started":1}
W20250110 02:51:19.324551 1879 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:19.223427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:44449 (local address 127.0.0.1:35138)
0110 02:51:19.322350 (+ 98923us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:19.322598 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":98726,"thread_start_us":10580,"threads_started":1}
I20250110 02:51:19.325415 1831 leader_election.cc:290] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.325704 1184 consensus_queue.cc:260] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.326082 1825 raft_consensus.cc:491] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.326267 1825 raft_consensus.cc:513] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.327140 1825 leader_election.cc:290] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.327713 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60486a3dbc5a48eaa26735b84ae3fd6f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.327921 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.328188 853 leader_election.cc:304] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.328724 1831 raft_consensus.cc:2743] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.328697 709 leader_election.cc:343] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.328919 915 raft_consensus.cc:3054] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:19.329113 853 leader_election.cc:343] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:19.329165 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef72e05f8ed4d46b759c7a04497653b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.329569 853 leader_election.cc:343] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:19.330327 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f01daf4c67462ba75e1d0cf56cef12" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.330946 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb7d0b9df3d48f0bb10eb6bc9b2cd6b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.331122 917 raft_consensus.cc:2463] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.331800 994 leader_election.cc:304] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.332252 1770 raft_consensus.cc:2743] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.332828 994 leader_election.cc:343] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.332978 994 leader_election.cc:304] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.333386 1770 raft_consensus.cc:2743] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.333873 994 leader_election.cc:304] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.334304 1770 raft_consensus.cc:2798] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.334426 1770 raft_consensus.cc:491] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.334445 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.334543 1770 raft_consensus.cc:3054] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:19.334829 711 leader_election.cc:343] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.334939 711 leader_election.cc:304] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.335111 1184 ts_tablet_manager.cc:1428] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250110 02:51:19.335300 1825 raft_consensus.cc:2743] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.336532 1184 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.337668 1770 raft_consensus.cc:513] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.339417 1770 leader_election.cc:290] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:19.339937 996 leader_election.cc:343] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
W20250110 02:51:19.340194 1888 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 02:51:19.243333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:44449 (local address 127.0.104.1:54159)
0110 02:51:19.325443 (+ 82110us) negotiation.cc:107] Waiting for socket to connect
0110 02:51:19.325585 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":81961,"thread_start_us":82368,"threads_started":1}
I20250110 02:51:19.340513 1181 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.340636 1181 ts_tablet_manager.cc:1397] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20250110 02:51:19.341516 1181 raft_consensus.cc:357] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.341697 1181 raft_consensus.cc:383] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.341780 1181 raft_consensus.cc:738] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.341958 1181 consensus_queue.cc:260] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.344178 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb7d0b9df3d48f0bb10eb6bc9b2cd6b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.344370 917 raft_consensus.cc:3054] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.345324 1181 ts_tablet_manager.cc:1428] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:19.345582 1181 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.347100 1181 tablet_bootstrap.cc:654] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.348310 917 raft_consensus.cc:2463] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.352216 1181 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.352351 1181 ts_tablet_manager.cc:1397] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250110 02:51:19.352983 1181 raft_consensus.cc:357] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.353181 1181 raft_consensus.cc:383] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.353255 1181 raft_consensus.cc:738] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.354079 994 leader_election.cc:304] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.354481 1770 raft_consensus.cc:2798] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.354624 1770 raft_consensus.cc:695] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.354882 1770 consensus_queue.cc:237] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.357161 1188 tablet_bootstrap.cc:492] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.357311 1188 ts_tablet_manager.cc:1397] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.038s
I20250110 02:51:19.357177 1181 consensus_queue.cc:260] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.358160 1181 ts_tablet_manager.cc:1428] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:19.358067 1188 raft_consensus.cc:357] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.358255 1188 raft_consensus.cc:383] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.358343 1188 raft_consensus.cc:738] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.358740 1181 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.358615 1188 consensus_queue.cc:260] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.359140 1188 ts_tablet_manager.cc:1428] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.360478 1181 tablet_bootstrap.cc:654] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.361405 1184 tablet_bootstrap.cc:654] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.362079 1181 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.362214 1188 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.362243 1181 ts_tablet_manager.cc:1397] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 02:51:19.363039 1181 raft_consensus.cc:357] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.363262 1181 raft_consensus.cc:383] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.363372 1181 raft_consensus.cc:738] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.363615 1181 consensus_queue.cc:260] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.363945 1188 tablet_bootstrap.cc:654] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.364302 1181 ts_tablet_manager.cc:1428] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:19.364581 1181 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.366045 1181 tablet_bootstrap.cc:654] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.369148 1188 tablet_bootstrap.cc:492] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.369345 1188 ts_tablet_manager.cc:1397] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250110 02:51:19.370034 1188 raft_consensus.cc:357] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.370347 1188 raft_consensus.cc:383] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.370465 1188 raft_consensus.cc:738] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.370759 1188 consensus_queue.cc:260] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.371279 1188 ts_tablet_manager.cc:1428] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:19.371620 1188 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.373167 1188 tablet_bootstrap.cc:654] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.375090 1804 tablet_loader.cc:96] loaded metadata for tablet 4aa78a9ef5eb443da0c8983a7190440b (table default.Table_97ca49cb31634f579d9d84e0d4fc8057 [id=6acef2fac29441ceac28a2721295136a])
I20250110 02:51:19.376741 1804 tablet_loader.cc:96] loaded metadata for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656])
I20250110 02:51:19.378293 1181 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.378469 1181 ts_tablet_manager.cc:1397] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250110 02:51:19.379278 1181 raft_consensus.cc:357] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.379628 1804 tablet_loader.cc:96] loaded metadata for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac])
I20250110 02:51:19.380520 1804 tablet_loader.cc:96] loaded metadata for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44])
I20250110 02:51:19.381160 1804 tablet_loader.cc:96] loaded metadata for tablet 4de67338a26f4febb5dd7611a72b4723 (table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8])
I20250110 02:51:19.379498 1181 raft_consensus.cc:383] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.381604 1181 raft_consensus.cc:738] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.381846 1181 consensus_queue.cc:260] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.382393 1181 ts_tablet_manager.cc:1428] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:19.382020 1804 tablet_loader.cc:96] loaded metadata for tablet 4e54ae5d7e864aff974a107f6762f446 (table default.Table_febf1d5b7a274c5fb59e10a180b4dd59 [id=23e8c0cf343748a092d750ea2ff7647a])
I20250110 02:51:19.382666 1181 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.383431 1804 tablet_loader.cc:96] loaded metadata for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_c8df69267ed3452192228c2106e18a47 [id=fa0cfecca12c4ddcade02cc7a553edf1])
I20250110 02:51:19.384325 1181 tablet_bootstrap.cc:654] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.388572 1804 tablet_loader.cc:96] loaded metadata for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989])
I20250110 02:51:19.389079 1804 tablet_loader.cc:96] loaded metadata for tablet 5a9d524fda0d4848b333df9e20a9f68b (table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da])
I20250110 02:51:19.390694 1804 tablet_loader.cc:96] loaded metadata for tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43])
I20250110 02:51:19.391391 1184 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.391507 1184 ts_tablet_manager.cc:1397] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.055s user 0.019s sys 0.019s
I20250110 02:51:19.392081 1184 raft_consensus.cc:357] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.392270 1184 raft_consensus.cc:383] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.392318 1184 raft_consensus.cc:738] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.392521 1184 consensus_queue.cc:260] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.392815 1831 raft_consensus.cc:491] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.392959 1831 raft_consensus.cc:513] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.393635 1831 leader_election.cc:290] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.395503 1804 tablet_loader.cc:96] loaded metadata for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39])
I20250110 02:51:19.395889 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.395877 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3352f8378b9846a3854086293f2d4e15" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.396059 774 raft_consensus.cc:2388] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:19.396067 1804 tablet_loader.cc:96] loaded metadata for tablet 6479d71a5142436b8c6ea2f76c221b33 (table default.Table_e5a7e456e4e143d1ac0f04b7cc5a1e14 [id=a09ade04ca6745f3899557ae0b416eaa])
I20250110 02:51:19.396502 853 leader_election.cc:304] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.396757 1184 ts_tablet_manager.cc:1428] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:19.397145 1831 raft_consensus.cc:3054] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.397801 1184 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.399176 1184 tablet_bootstrap.cc:654] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.401805 1831 raft_consensus.cc:2743] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.403712 1804 tablet_loader.cc:96] loaded metadata for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_e8d197a297e647b5ad3b516ff8054704 [id=f733132df99b481ea5a2117fe65a7fc4])
I20250110 02:51:19.405645 1804 tablet_loader.cc:96] loaded metadata for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_68d7c1c889464468adcd9153caef7ecc [id=497a56d44a114093934dbb9fde1b9f8a])
I20250110 02:51:19.411015 1770 raft_consensus.cc:491] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.411432 1770 raft_consensus.cc:513] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.412992 1770 leader_election.cc:290] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.417317 1804 tablet_loader.cc:96] loaded metadata for tablet 6694671f422c4be7b545304a38b17edd (table default.Table_68a6c97dd17c47b28f987d6e8e778e37 [id=4d8aac2aeda44055ab88ed9f82e8efeb])
W20250110 02:51:19.418190 996 leader_election.cc:343] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:19.418519 1849 consensus_queue.cc:1035] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.404552 1188 tablet_bootstrap.cc:492] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.425009 1188 ts_tablet_manager.cc:1397] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.006s
I20250110 02:51:19.426014 1804 tablet_loader.cc:96] loaded metadata for tablet 67294cdecffc4ad4b35e000a3e01c9be (table default.Table_c5d78661dad44d9f85ba6c91486185de [id=a6a89e8f33c74e61b95de2207ddbd585])
I20250110 02:51:19.426069 1188 raft_consensus.cc:357] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.426357 1188 raft_consensus.cc:383] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.426515 1188 raft_consensus.cc:738] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.426787 1188 consensus_queue.cc:260] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.427398 1188 ts_tablet_manager.cc:1428] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:19.427827 1188 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.430022 1188 tablet_bootstrap.cc:654] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.435441 1831 raft_consensus.cc:491] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.435693 1831 raft_consensus.cc:513] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.436987 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.438107 1831 leader_election.cc:290] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.438405 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6757b7ecc514489a0fa7e1ff18206bd" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.439000 853 leader_election.cc:304] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.439865 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ace37b96844328ccdb821dad3c5ea" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.441586 1832 raft_consensus.cc:2743] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.441649 994 leader_election.cc:343] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.441771 994 leader_election.cc:304] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.441807 1832 raft_consensus.cc:491] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.441941 1832 raft_consensus.cc:513] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.442225 1849 raft_consensus.cc:2743] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.442807 1832 leader_election.cc:290] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.444063 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb5412b78104d8085878c58b191ab27" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.444535 853 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.444754 853 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 1fb5412b78104d8085878c58b191ab27
I20250110 02:51:19.444864 853 leader_election.cc:304] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.445356 1832 raft_consensus.cc:2743] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.446142 1825 raft_consensus.cc:491] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.446307 1825 raft_consensus.cc:513] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.447073 1825 leader_election.cc:290] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.447785 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.448139 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.448652 711 leader_election.cc:343] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.448797 709 leader_election.cc:343] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.448925 709 leader_election.cc:304] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.449191 1825 raft_consensus.cc:2743] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.450286 1188 tablet_bootstrap.cc:492] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.450515 1188 ts_tablet_manager.cc:1397] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250110 02:51:19.451182 1184 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.451380 1184 ts_tablet_manager.cc:1397] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.019s
I20250110 02:51:19.451318 1188 raft_consensus.cc:357] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.451506 1188 raft_consensus.cc:383] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.451599 1188 raft_consensus.cc:738] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.451803 1188 consensus_queue.cc:260] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.452530 1188 ts_tablet_manager.cc:1428] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:19.453544 1188 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.455143 1184 raft_consensus.cc:357] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.455372 1184 raft_consensus.cc:383] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.455467 1184 raft_consensus.cc:738] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.455736 1184 consensus_queue.cc:260] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.456105 1188 tablet_bootstrap.cc:654] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.460762 1184 ts_tablet_manager.cc:1428] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:19.462397 1184 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.462936 1181 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.463204 1181 ts_tablet_manager.cc:1397] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.042s
I20250110 02:51:19.463833 1804 tablet_loader.cc:96] loaded metadata for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f])
I20250110 02:51:19.464318 1184 tablet_bootstrap.cc:654] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.464447 1181 raft_consensus.cc:357] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.464823 1181 raft_consensus.cc:383] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.465018 1181 raft_consensus.cc:738] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.465626 1804 tablet_loader.cc:96] loaded metadata for tablet 6e0588e5e5714cc5bd1c73665e5645b1 (table default.Table_ca9f3a9eeaa845498cf76982cc9b59fd [id=bda31fe0278a4b9a95eedb85d71b2eff])
I20250110 02:51:19.465413 1181 consensus_queue.cc:260] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.466329 1181 ts_tablet_manager.cc:1428] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:19.467147 1181 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.470026 1181 tablet_bootstrap.cc:654] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.473407 1184 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.473634 1184 ts_tablet_manager.cc:1397] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20250110 02:51:19.474570 1184 raft_consensus.cc:357] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.474916 1184 raft_consensus.cc:383] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.475221 1184 raft_consensus.cc:738] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.475562 1184 consensus_queue.cc:260] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.477449 1184 ts_tablet_manager.cc:1428] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:19.478658 1184 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.480449 1184 tablet_bootstrap.cc:654] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.483011 1804 tablet_loader.cc:96] loaded metadata for tablet 6e758fcd046f4196a3735d4b731544e6 (table default.Table_f8bbe9b2e40f4749b881e4be2e8ed33a [id=90471593be8d4c04b0ac2d50a96966e2])
I20250110 02:51:19.483307 1181 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.483557 1181 ts_tablet_manager.cc:1397] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20250110 02:51:19.485157 1804 tablet_loader.cc:96] loaded metadata for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246])
I20250110 02:51:19.485756 1181 raft_consensus.cc:357] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.486697 1181 raft_consensus.cc:383] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.487083 1181 raft_consensus.cc:738] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.488037 1181 consensus_queue.cc:260] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.490427 1181 ts_tablet_manager.cc:1428] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:19.491881 1770 raft_consensus.cc:491] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.492044 1770 raft_consensus.cc:513] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.492388 1181 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.492861 1770 leader_election.cc:290] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.493999 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.494202 776 raft_consensus.cc:2463] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.494966 996 leader_election.cc:304] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:19.495446 1680 raft_consensus.cc:2798] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.495563 1680 raft_consensus.cc:491] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.495695 1680 raft_consensus.cc:3054] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.500056 1181 tablet_bootstrap.cc:654] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.503322 1680 raft_consensus.cc:513] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.510699 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:19.510888 776 raft_consensus.cc:3054] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.512295 1849 consensus_queue.cc:1035] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.513404 776 raft_consensus.cc:2463] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.513722 1804 tablet_loader.cc:96] loaded metadata for tablet 71534cc7dd014bc1afb365de96977f8c (table default.Table_70a31a6f02a443d68091578355b71adc [id=4946b19488484f39ad47b1118ce03359])
I20250110 02:51:19.513793 1831 raft_consensus.cc:491] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.513885 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.513967 1831 raft_consensus.cc:513] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.514094 1804 tablet_loader.cc:96] loaded metadata for tablet 718d3add9d0a4e4b8633f2c977e316e6 (table default.Table_c99830786e0e4b7b956e604dd3a5fcc0 [id=2af0614c37b44ebb9baa17b28b82f381])
I20250110 02:51:19.514115 915 raft_consensus.cc:2463] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.514276 996 leader_election.cc:304] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:19.514564 1804 tablet_loader.cc:96] loaded metadata for tablet 71ffa250a3554a7b8f3a9c34ee7e4674 (table default.Table_6f9765add6c947e8b8154e36e6d358dc [id=7badf15e8d1543209edcffa451e3b48c])
I20250110 02:51:19.514647 1849 raft_consensus.cc:2798] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.514636 1831 leader_election.cc:290] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.514760 1849 raft_consensus.cc:695] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.515029 1849 consensus_queue.cc:237] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.515398 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e534bf66f044b88d8cda0f3a1228f4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.515501 1804 tablet_loader.cc:96] loaded metadata for tablet 7564208da99449cd97338a41b263e750 (table default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d])
I20250110 02:51:19.515587 1061 raft_consensus.cc:2463] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.515872 1804 tablet_loader.cc:96] loaded metadata for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e])
I20250110 02:51:19.515981 853 leader_election.cc:304] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.516208 1680 leader_election.cc:290] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.516295 1831 raft_consensus.cc:2798] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.516396 1831 raft_consensus.cc:491] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.516482 1831 raft_consensus.cc:3054] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.519393 1831 raft_consensus.cc:513] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:19.520808 853 leader_election.cc:343] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:19.521733 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e534bf66f044b88d8cda0f3a1228f4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.521945 1061 raft_consensus.cc:3054] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.523108 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b2297377bc4e5095826e30523eede7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.523280 917 raft_consensus.cc:3054] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.524452 1061 raft_consensus.cc:2463] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.524919 853 leader_election.cc:304] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.525233 1832 raft_consensus.cc:2798] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.525346 1832 raft_consensus.cc:695] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.525589 1832 consensus_queue.cc:237] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.525935 917 raft_consensus.cc:2463] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:19.527150 853 leader_election.cc:343] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:19.528028 1831 leader_election.cc:290] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.540865 1832 raft_consensus.cc:491] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.541081 1832 raft_consensus.cc:513] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.542421 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e183122d13a4c38988bf2fe6f7a3694" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.542629 1061 raft_consensus.cc:2463] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.543092 853 leader_election.cc:304] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.543417 1831 raft_consensus.cc:2798] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.543503 1831 raft_consensus.cc:491] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.543602 1831 raft_consensus.cc:3054] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.546188 1831 raft_consensus.cc:513] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.547520 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e183122d13a4c38988bf2fe6f7a3694" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.547745 1061 raft_consensus.cc:3054] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.548713 1181 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.548889 1181 ts_tablet_manager.cc:1397] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.004s
I20250110 02:51:19.549762 1181 raft_consensus.cc:357] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.549947 1181 raft_consensus.cc:383] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.550027 1181 raft_consensus.cc:738] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.551112 1061 raft_consensus.cc:2463] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.551591 853 leader_election.cc:304] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.551903 1726 raft_consensus.cc:2798] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.550242 1181 consensus_queue.cc:260] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.553150 1181 ts_tablet_manager.cc:1428] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:19.554116 1181 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:19.554227 853 leader_election.cc:343] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:19.554546 1831 leader_election.cc:290] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.555780 1804 tablet_loader.cc:96] loaded metadata for tablet 7752241af2fb4f959f00252faee87e98 (table default.Table_410dc5667b5446fca70ebd92193bbc32 [id=1d5bb5a0ce4e4f91babd89611af071ac])
I20250110 02:51:19.556314 1804 tablet_loader.cc:96] loaded metadata for tablet 78c0808330e848249e1d7eb71ab0d362 (table default.Table_375ea34c1b6d471ca632286504f259f0 [id=5d041eb06ad54278b7a327c69bbae96a])
I20250110 02:51:19.556797 1804 tablet_loader.cc:96] loaded metadata for tablet 78d9140c04a14afc96d43f3bc93488e9 (table default.Table_5131fbd504e04f50b92a6a1fb6d0c055 [id=7c940859b1954178bfcb14fb882668f6])
I20250110 02:51:19.557260 1804 tablet_loader.cc:96] loaded metadata for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0])
W20250110 02:51:19.557296 853 leader_election.cc:343] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:19.557725 1804 tablet_loader.cc:96] loaded metadata for tablet 7b3d6e0c043449688ed6450cb2a981a3 (table default.Table_b064f366436b4b72bc8428d0bbb0c3df [id=5b2f66225983487abf161e772efb6ac2])
I20250110 02:51:19.558249 1181 tablet_bootstrap.cc:654] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.557613 1832 leader_election.cc:290] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.565084 1184 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.565224 1184 ts_tablet_manager.cc:1397] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.017s
I20250110 02:51:19.565945 1184 raft_consensus.cc:357] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.566143 1184 raft_consensus.cc:383] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.566203 1184 raft_consensus.cc:738] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.566511 1181 tablet_bootstrap.cc:492] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.566437 1184 consensus_queue.cc:260] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.566807 1181 ts_tablet_manager.cc:1397] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20250110 02:51:19.567737 1181 raft_consensus.cc:357] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.568104 1184 ts_tablet_manager.cc:1428] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:19.568058 1770 raft_consensus.cc:491] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.568857 1726 raft_consensus.cc:695] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.568850 1770 raft_consensus.cc:513] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.569141 1726 consensus_queue.cc:237] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.569442 1770 leader_election.cc:290] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.570106 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66603a0ee8c4082807c5a6aabe7e339" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.570726 994 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.571542 1181 raft_consensus.cc:383] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.571159 1184 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:19.571610 996 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:19.571861 996 leader_election.cc:304] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.572294 1770 raft_consensus.cc:2743] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.572496 1181 raft_consensus.cc:738] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.572892 1181 consensus_queue.cc:260] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.573359 1184 tablet_bootstrap.cc:654] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.574551 1181 ts_tablet_manager.cc:1428] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:19.575875 1181 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.580633 1181 tablet_bootstrap.cc:654] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.585345 1184 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.586443 1804 tablet_loader.cc:96] loaded metadata for tablet 7c2c7a313ba94e5ea85e45276658ba04 (table default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20])
I20250110 02:51:19.587002 1804 tablet_loader.cc:96] loaded metadata for tablet 7c55788eeddb47aa8a5893703c4d5e67 (table default.Table_27ec9682796f420ebd7920d531fce2fa [id=0b98c3d632294ec4bfa3fe55e718afac])
I20250110 02:51:19.589710 1184 ts_tablet_manager.cc:1397] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20250110 02:51:19.591229 1188 tablet_bootstrap.cc:492] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.591363 1188 ts_tablet_manager.cc:1397] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.138s user 0.020s sys 0.028s
I20250110 02:51:19.591248 1184 raft_consensus.cc:357] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.591651 1184 raft_consensus.cc:383] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.591774 1184 raft_consensus.cc:738] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.592000 1188 raft_consensus.cc:357] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.592180 1188 raft_consensus.cc:383] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.592255 1188 raft_consensus.cc:738] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.592245 1184 consensus_queue.cc:260] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.592602 1188 consensus_queue.cc:260] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.593211 1188 ts_tablet_manager.cc:1428] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:19.593220 1184 ts_tablet_manager.cc:1428] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250110 02:51:19.593570 1188 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.593883 1184 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.595422 1188 tablet_bootstrap.cc:654] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.601609 1184 tablet_bootstrap.cc:654] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.605849 1804 tablet_loader.cc:96] loaded metadata for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb])
I20250110 02:51:19.606329 1804 tablet_loader.cc:96] loaded metadata for tablet 7efa857721c34003ad91912ef2d7b381 (table default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c])
I20250110 02:51:19.606974 1916 raft_consensus.cc:491] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.607141 1916 raft_consensus.cc:513] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.607695 1804 tablet_loader.cc:96] loaded metadata for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462])
I20250110 02:51:19.607975 1916 leader_election.cc:290] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.608119 1804 tablet_loader.cc:96] loaded metadata for tablet 8162ceb17c714614b34beb33a7633ff0 (table default.Table_627ff2f24b97485a9f948b72f8b39daf [id=3e8497573a584513ab1df4ba5ed27993])
I20250110 02:51:19.608651 1804 tablet_loader.cc:96] loaded metadata for tablet 818dcbf83303412d97b097913bcd36c3 (table default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91])
I20250110 02:51:19.609094 1804 tablet_loader.cc:96] loaded metadata for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb])
I20250110 02:51:19.609799 1188 tablet_bootstrap.cc:492] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.609951 1188 ts_tablet_manager.cc:1397] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.008s
I20250110 02:51:19.609925 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4422eca5604d149e73c2a669d07af0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.610209 1061 raft_consensus.cc:2463] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.610671 1188 raft_consensus.cc:357] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.610929 1188 raft_consensus.cc:383] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.611012 1188 raft_consensus.cc:738] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.611068 853 leader_election.cc:304] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.611253 1188 consensus_queue.cc:260] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.611429 853 leader_election.cc:343] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:19.611498 1832 raft_consensus.cc:2798] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.611567 1832 raft_consensus.cc:491] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.611644 1832 raft_consensus.cc:3054] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.614250 1832 raft_consensus.cc:513] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.616675 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4422eca5604d149e73c2a669d07af0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.616888 1061 raft_consensus.cc:3054] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.617396 1184 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.617600 1184 ts_tablet_manager.cc:1397] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20250110 02:51:19.619148 1184 raft_consensus.cc:357] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.619412 1184 raft_consensus.cc:383] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.619499 1184 raft_consensus.cc:738] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.619573 1061 raft_consensus.cc:2463] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.619887 1184 consensus_queue.cc:260] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.620831 1184 ts_tablet_manager.cc:1428] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:19.621738 1184 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.626411 1184 tablet_bootstrap.cc:654] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.632927 1832 leader_election.cc:290] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.633304 1188 ts_tablet_manager.cc:1428] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250110 02:51:19.634598 1188 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.635176 1804 tablet_loader.cc:96] loaded metadata for tablet 82bd4307e5fe4f61b9308005cf3446e1 (table default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2])
I20250110 02:51:19.635551 1804 tablet_loader.cc:96] loaded metadata for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_6a8868f4b22043c193efdd181759b4f4 [id=fd05c3a951c64cbf81dfd168728165f4])
I20250110 02:51:19.635986 1804 tablet_loader.cc:96] loaded metadata for tablet 8600a637306741dd8e1e7f4964ff4bc8 (table default.Table_eb40720cee0c4ef08e2c5a49f1374c8e [id=40f4c299cde64d02bd2a420d11b6765f])
I20250110 02:51:19.636828 1825 raft_consensus.cc:491] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.636982 1825 raft_consensus.cc:513] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.637748 1825 leader_election.cc:290] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.638265 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.638576 1184 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.638810 1184 ts_tablet_manager.cc:1397] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
W20250110 02:51:19.638885 711 leader_election.cc:343] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.639758 1181 tablet_bootstrap.cc:492] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.639946 1181 ts_tablet_manager.cc:1397] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.025s
I20250110 02:51:19.640414 1184 raft_consensus.cc:357] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.640787 1184 raft_consensus.cc:383] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.640882 1184 raft_consensus.cc:738] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.641038 853 leader_election.cc:304] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.641116 1804 tablet_loader.cc:96] loaded metadata for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d])
I20250110 02:51:19.641254 1184 consensus_queue.cc:260] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:19.641553 853 leader_election.cc:343] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:19.641832 1832 raft_consensus.cc:2798] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.641968 1832 raft_consensus.cc:695] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.642216 1184 ts_tablet_manager.cc:1428] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:19.642196 1832 consensus_queue.cc:237] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.643944 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.644840 1188 tablet_bootstrap.cc:654] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.645679 1916 raft_consensus.cc:491] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.645864 1916 raft_consensus.cc:513] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.646435 1916 leader_election.cc:290] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:19.647377 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
W20250110 02:51:19.647534 709 leader_election.cc:343] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.647643 1184 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.647643 709 leader_election.cc:304] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.648988 1825 raft_consensus.cc:2743] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.649438 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
I20250110 02:51:19.649570 853 leader_election.cc:304] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.649958 1916 raft_consensus.cc:2743] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.650781 1916 raft_consensus.cc:491] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.650949 1916 raft_consensus.cc:513] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.651621 1916 leader_election.cc:290] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.652347 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8f247c862e4cb68452e230857ee6b0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.652644 1061 raft_consensus.cc:2463] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
W20250110 02:51:19.653542 853 leader_election.cc:343] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:19.653702 853 leader_election.cc:304] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.654119 1916 raft_consensus.cc:2798] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.654194 1916 raft_consensus.cc:491] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.654271 1916 raft_consensus.cc:3054] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.654297 1804 tablet_loader.cc:96] loaded metadata for tablet 89b5554630004cc6bb2574863f07cbd4 (table default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c [id=103f9b0512f343a29e3d8ba6de6071ee])
I20250110 02:51:19.656926 1916 raft_consensus.cc:513] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.657626 1916 leader_election.cc:290] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.657481 1804 tablet_loader.cc:96] loaded metadata for tablet 8b255096800d4cd58324e2c06ff1a44c (table default.Table_c0c89321da6c41cbbf765c7562f60e95 [id=866547595c5d4cce9221971e917542eb])
I20250110 02:51:19.657748 1184 tablet_bootstrap.cc:654] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.658357 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da8f247c862e4cb68452e230857ee6b0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.658627 1061 raft_consensus.cc:3054] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.659682 1804 tablet_loader.cc:96] loaded metadata for tablet 8bc32061409246a2b280dde4f9180a8e (table default.Table_73259c80bde24fd3a2440bdce73bd88e [id=0afe16dcfb0c4ef0897ceb5fb2993d7b])
I20250110 02:51:19.661262 1061 raft_consensus.cc:2463] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.661445 1804 tablet_loader.cc:96] loaded metadata for tablet 8bc89802926543738cb04c75821ab693 (table default.Table_e85cbad8a6e24bf59145c42e3b94ebf1 [id=056af6d890614cae954fa6f9e3f7f6d7])
W20250110 02:51:19.662976 853 leader_election.cc:343] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:19.663223 853 leader_election.cc:304] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.663342 1770 raft_consensus.cc:491] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.663477 1770 raft_consensus.cc:513] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.663925 1916 raft_consensus.cc:2798] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.664124 1916 raft_consensus.cc:695] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.664125 1770 leader_election.cc:290] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.665275 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad0d561a8ef498ba13bc813d4cd82fa" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.663688 1804 tablet_loader.cc:96] loaded metadata for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144])
I20250110 02:51:19.665462 917 raft_consensus.cc:2463] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:19.666047 996 leader_election.cc:343] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:19.666538 994 leader_election.cc:304] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.666798 1770 raft_consensus.cc:2798] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.666898 1770 raft_consensus.cc:491] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.666986 1770 raft_consensus.cc:3054] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.664402 1916 consensus_queue.cc:237] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.669013 1680 raft_consensus.cc:491] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.669168 1680 raft_consensus.cc:513] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.669600 1770 raft_consensus.cc:513] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.670185 1770 leader_election.cc:290] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.669689 1680 leader_election.cc:290] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.670564 1804 tablet_loader.cc:96] loaded metadata for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 (table default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9])
W20250110 02:51:19.671001 996 leader_election.cc:343] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc
W20250110 02:51:19.671180 996 leader_election.cc:343] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:19.671737 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad0d561a8ef498ba13bc813d4cd82fa" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.671913 915 raft_consensus.cc:3054] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.672709 1804 tablet_loader.cc:96] loaded metadata for tablet 9395aa9b39d04ac9b6cecf106bc91499 (table default.Table_adb6100fd2ea489590831f0508f53e3e [id=811a01d9de1a4185a876251bd8d10512])
I20250110 02:51:19.673243 1804 tablet_loader.cc:96] loaded metadata for tablet 9497c76dcffa45268f705ca6423db41c (table default.Table_8835f1dac47741f69e9a48aed136cde0 [id=e8a9e62e18bf419e807e689558db87ea])
I20250110 02:51:19.674053 1804 tablet_loader.cc:96] loaded metadata for tablet 971f97a3be9b49958b1218a275e126d4 (table default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208])
I20250110 02:51:19.640606 1181 raft_consensus.cc:357] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.674472 1181 raft_consensus.cc:383] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.674580 1181 raft_consensus.cc:738] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.674527 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e72ed2d31a8467f8c092b48d593c1cc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.674710 917 raft_consensus.cc:2463] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.674808 1181 consensus_queue.cc:260] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.675446 915 raft_consensus.cc:2463] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.676275 994 leader_election.cc:304] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.677590 1184 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.677706 1184 ts_tablet_manager.cc:1397] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20250110 02:51:19.678470 1184 raft_consensus.cc:357] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.678673 1184 raft_consensus.cc:383] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.678726 1184 raft_consensus.cc:738] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.678905 1184 consensus_queue.cc:260] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.679391 1184 ts_tablet_manager.cc:1428] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:19.679605 1184 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.680503 1188 tablet_bootstrap.cc:492] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.680635 1188 ts_tablet_manager.cc:1397] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.005s
I20250110 02:51:19.681074 1184 tablet_bootstrap.cc:654] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.681345 994 leader_election.cc:304] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.681304 1188 raft_consensus.cc:357] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.681466 1188 raft_consensus.cc:383] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.681560 1188 raft_consensus.cc:738] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.681767 1188 consensus_queue.cc:260] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.681900 1804 tablet_loader.cc:96] loaded metadata for tablet 9d3e52914a424cb3a8bf422b3a56fb22 (table default.Table_f9ee107db2ba4f2d9b62e9459ea41457 [id=0e2933f55f834e8baa0a11bda1b4acb4])
I20250110 02:51:19.682210 1804 tablet_loader.cc:96] loaded metadata for tablet 9da4d8583ce74605a80da29ac030692a (table default.Table_34ce8eea83364d1ca7d8cfa100115b45 [id=4696b770f7514e2f9d5aab8e16ad9eae])
I20250110 02:51:19.682229 1188 ts_tablet_manager.cc:1428] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:19.682577 1188 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.682574 1804 tablet_loader.cc:96] loaded metadata for tablet 9e35fe84c1e04d3eb295a6d8d5d40ea4 (table default.Table_8cf4c26686b84eb59c11d17010f8a679 [id=d37919aec6fc44ba9c1800b84d5fe13a])
I20250110 02:51:19.683095 1804 tablet_loader.cc:96] loaded metadata for tablet 9f94ea0136d24be2bd5b3dac9b7b16b7 (table default.Table_2e65de9dd8c84b94aa607fce39e93fb9 [id=f08b0741c62e46ddbfe72684657fe034])
I20250110 02:51:19.683411 1804 tablet_loader.cc:96] loaded metadata for tablet a0ddab9e847445999ec97eac4352d9c2 (table default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9])
I20250110 02:51:19.684409 1188 tablet_bootstrap.cc:654] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.685201 1680 raft_consensus.cc:2798] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.685297 1680 raft_consensus.cc:491] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.685393 1680 raft_consensus.cc:3054] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.687582 1759 raft_consensus.cc:2798] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.687726 1759 raft_consensus.cc:695] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.687930 1680 raft_consensus.cc:513] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.688017 1759 consensus_queue.cc:237] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.689201 1680 leader_election.cc:290] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.689576 1832 consensus_queue.cc:1035] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:19.689733 996 leader_election.cc:343] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:19.692766 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e72ed2d31a8467f8c092b48d593c1cc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.692930 915 raft_consensus.cc:3054] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.696230 915 raft_consensus.cc:2463] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.697196 994 leader_election.cc:304] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.697685 1770 raft_consensus.cc:2798] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.697815 1770 raft_consensus.cc:695] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.698069 1770 consensus_queue.cc:237] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.700938 1181 ts_tablet_manager.cc:1428] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250110 02:51:19.701169 1770 raft_consensus.cc:491] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.701306 1770 raft_consensus.cc:513] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.701793 1770 leader_election.cc:290] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.701951 1804 tablet_loader.cc:96] loaded metadata for tablet a254ed6c239249c6982429eb5e142031 (table default.Table_7cebf0b5d59d4f068fc8dce07001bd12 [id=f686f55e175d4a0bbdaa0d8a98d63791])
W20250110 02:51:19.702641 996 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:19.703174 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ae1c19d2dc4778a1cebddbd505f907" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.703372 915 raft_consensus.cc:2463] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:19.703691 994 leader_election.cc:304] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.704061 1770 raft_consensus.cc:2798] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.704146 1770 raft_consensus.cc:491] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.704262 1770 raft_consensus.cc:3054] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.704452 1804 tablet_loader.cc:96] loaded metadata for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b])
I20250110 02:51:19.705097 1804 tablet_loader.cc:96] loaded metadata for tablet a7397e41f7374e9fb1621dbcf30af703 (table default.Table_94cf6a20190647dd87d3616b069b0243 [id=a91e19e65fd74454b4b4d339a5cbe806])
I20250110 02:51:19.706869 1770 raft_consensus.cc:513] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.707746 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ae1c19d2dc4778a1cebddbd505f907" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:19.707944 917 raft_consensus.cc:3054] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.708410 1804 tablet_loader.cc:96] loaded metadata for tablet aa2b5a305db34fedbbbf0d666e6c28cb (table default.Table_a993dbbdb6714184867df68d17523ca7 [id=8cf03d1b06f24347a2f8c4132f135d75])
I20250110 02:51:19.709393 1804 tablet_loader.cc:96] loaded metadata for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_09dd77d568b8468cbad2375415ecb197 [id=c6fdcf73aa2a471faeaca5a4a8f70258])
I20250110 02:51:19.709498 1188 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.709662 1188 ts_tablet_manager.cc:1397] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20250110 02:51:19.709915 1804 tablet_loader.cc:96] loaded metadata for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878])
I20250110 02:51:19.710316 1804 tablet_loader.cc:96] loaded metadata for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba])
I20250110 02:51:19.710366 1188 raft_consensus.cc:357] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.710598 1188 raft_consensus.cc:383] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.710677 1188 raft_consensus.cc:738] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.710923 917 raft_consensus.cc:2463] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:19.711107 1188 consensus_queue.cc:260] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.711762 1188 ts_tablet_manager.cc:1428] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.712070 1188 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.712244 994 leader_election.cc:304] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.713702 1188 tablet_bootstrap.cc:654] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.714207 996 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:19.714417 1770 leader_election.cc:290] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.714726 1770 raft_consensus.cc:2798] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.715230 1770 raft_consensus.cc:695] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:19.716048 1770 consensus_queue.cc:237] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.717685 1804 tablet_loader.cc:96] loaded metadata for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433])
I20250110 02:51:19.718139 1804 tablet_loader.cc:96] loaded metadata for tablet b010e9c549214937889fa281542e752e (table default.Table_62d79207c4534af2becc6ccc6da65f1e [id=8e8cecbb687a48009172ae19382c9b80])
I20250110 02:51:19.718609 1804 tablet_loader.cc:96] loaded metadata for tablet b12ce7a0dc4f4f3586a2f918d31234e1 (table default.Table_dd0bd0decd37446f891e12d29ab4955b [id=92d9235395b24e1eb75141de12dbe3e7])
I20250110 02:51:19.719424 1804 tablet_loader.cc:96] loaded metadata for tablet b25bbf8aa71e478587093627c715310f (table default.Table_a3ba2bc51a3b4fc6b6451349e08b106d [id=14dfd1504b024d6bbd154720e1e8c43f])
I20250110 02:51:19.719671 1181 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.721094 1181 tablet_bootstrap.cc:654] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.722448 1804 tablet_loader.cc:96] loaded metadata for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326])
I20250110 02:51:19.725005 1804 tablet_loader.cc:96] loaded metadata for tablet b48fe2d061da4e729c8704af4fc83f7a (table default.Table_6155a05a8ab843e8a0ad9f115a26d4e9 [id=2fe86361000e43eda29c3a39e78beeee])
I20250110 02:51:19.725272 1184 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.725402 1184 ts_tablet_manager.cc:1397] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20250110 02:51:19.725504 1804 tablet_loader.cc:96] loaded metadata for tablet b4c2a19225ed477281b7ce60ae25b084 (table default.Table_3f51e7f8aaa7470fbeeb5459695efab1 [id=4b8aabec28ba49388402a7d16de036c5])
I20250110 02:51:19.725852 1804 tablet_loader.cc:96] loaded metadata for tablet b4f1782c46a44ed091f4c533226c71f2 (table default.Table_18eb7bac666944ddaf84da77edd4462b [id=5d3953c49f6f46b29e8ea2cd1f72f8d0])
I20250110 02:51:19.726248 1804 tablet_loader.cc:96] loaded metadata for tablet b5134444da58427aa97e166e44a3a648 (table default.Table_fe9783d91cae4578b6f0e7a6cc61bc84 [id=c758d6f1cb1e4e48bee42c36a5df0e04])
I20250110 02:51:19.726181 1184 raft_consensus.cc:357] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.726358 1184 raft_consensus.cc:383] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.726437 1184 raft_consensus.cc:738] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.726801 1804 tablet_loader.cc:96] loaded metadata for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf])
I20250110 02:51:19.726864 1184 consensus_queue.cc:260] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.727272 1804 tablet_loader.cc:96] loaded metadata for tablet b74beb77505f472fb6b9e4a30bfe6582 (table default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6])
I20250110 02:51:19.728328 1804 tablet_loader.cc:96] loaded metadata for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_35002d6ab8564fe1b93188d7ce973d3f [id=ed3a9f13f7794c3f9ee6917bf6ecff25])
I20250110 02:51:19.728778 1804 tablet_loader.cc:96] loaded metadata for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd])
I20250110 02:51:19.729182 1804 tablet_loader.cc:96] loaded metadata for tablet bc8444c1381a45d6b2fee30da971bd47 (table default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253])
I20250110 02:51:19.729791 1804 tablet_loader.cc:96] loaded metadata for tablet bd238fa1922d456789b59bf5cee52327 (table default.Table_09cf33925be04c35b8a1304633c52217 [id=6f90cd9154c3419b8d3fd88395270251])
I20250110 02:51:19.730222 1804 tablet_loader.cc:96] loaded metadata for tablet bd6d4c4631994a5792200ba4d4c906f0 (table default.Table_742e66ce36c0463a978cb00de2f01135 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff])
I20250110 02:51:19.730252 1184 ts_tablet_manager.cc:1428] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250110 02:51:19.730687 1804 tablet_loader.cc:96] loaded metadata for tablet be5ac15f4e774b7e854e4c5b4737b591 (table default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d])
I20250110 02:51:19.731118 1804 tablet_loader.cc:96] loaded metadata for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701])
I20250110 02:51:19.731791 1804 tablet_loader.cc:96] loaded metadata for tablet c09235548b5d4e10a01cbe05d1c94841 (table default.Table_26b5c54688724d26b6bf4f3587158e9c [id=9e880e06a6d648b7bc44b651084a14a3])
I20250110 02:51:19.732097 1184 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.732570 1804 tablet_loader.cc:96] loaded metadata for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd])
I20250110 02:51:19.735415 1188 tablet_bootstrap.cc:492] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.735427 1181 tablet_bootstrap.cc:492] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.735553 1188 ts_tablet_manager.cc:1397] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250110 02:51:19.735570 1181 ts_tablet_manager.cc:1397] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20250110 02:51:19.736284 1188 raft_consensus.cc:357] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.736266 1181 raft_consensus.cc:357] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.736455 1188 raft_consensus.cc:383] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.736485 1181 raft_consensus.cc:383] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.736539 1188 raft_consensus.cc:738] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.736563 1181 raft_consensus.cc:738] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.736827 1188 consensus_queue.cc:260] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.737553 1188 ts_tablet_manager.cc:1428] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.737833 1188 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.739223 1181 consensus_queue.cc:260] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.739593 1188 tablet_bootstrap.cc:654] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.739864 1181 ts_tablet_manager.cc:1428] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:19.740622 1804 tablet_loader.cc:96] loaded metadata for tablet c1ade70d7fe34776bef3c5853f8dde74 (table default.Table_c7fa3a12842444fa9aa1758777787789 [id=88412101305b4d0a8dea71c08ff391aa])
I20250110 02:51:19.741760 1804 tablet_loader.cc:96] loaded metadata for tablet c52f74fe375440a28139c27f3f868970 (table default.Table_77792a05daee44cfaaebbe693ef3cd65 [id=f1926d48a14c45699ed056c8b1818b4e])
I20250110 02:51:19.742269 1184 tablet_bootstrap.cc:654] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.745090 1804 tablet_loader.cc:96] loaded metadata for tablet c975136880034808b58276e06fbd94f9 (table default.Table_3b60f335e3574a5bbf7ab20a93a64bed [id=b00159b02a3a4c2d877a7e8c7bf1fdeb])
I20250110 02:51:19.745519 1804 tablet_loader.cc:96] loaded metadata for tablet ca45a5c0b3cb4b53862e51b11c4d3a00 (table default.Table_60665b68321047d3a87e132a42f6c058 [id=8c72b8cb59bc4abdb98913a891ef023a])
I20250110 02:51:19.746412 1804 tablet_loader.cc:96] loaded metadata for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc])
I20250110 02:51:19.753492 1804 tablet_loader.cc:96] loaded metadata for tablet d1f3e8cc6ac4442690b9d062bf1c2dc4 (table default.Table_bb1c63b5d7ee453a999514a777e71496 [id=ab05074705fa48438c82fd089d97be75])
I20250110 02:51:19.754212 1804 tablet_loader.cc:96] loaded metadata for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c])
I20250110 02:51:19.754673 1804 tablet_loader.cc:96] loaded metadata for tablet d424b4da66184bd19eccefad80621516 (table default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6])
I20250110 02:51:19.757675 1188 tablet_bootstrap.cc:492] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.757840 1188 ts_tablet_manager.cc:1397] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20250110 02:51:19.758703 1804 tablet_loader.cc:96] loaded metadata for tablet d5f56778a0244378ad8499a4ceef3510 (table default.Table_a9587fbd6e924c49b1a9a1189bff5d63 [id=4e31223c509f46eba782e7c34f293c18])
I20250110 02:51:19.758702 1188 raft_consensus.cc:357] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.758910 1188 raft_consensus.cc:383] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.758991 1188 raft_consensus.cc:738] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.759204 1188 consensus_queue.cc:260] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.759328 1804 tablet_loader.cc:96] loaded metadata for tablet d637597047ab4738a17eca2f5409459a (table default.Table_fdcd90a5deda47968fc8ba979edbf5a2 [id=bae70feea8bb4af3b1e7601399e19534])
I20250110 02:51:19.759711 1188 ts_tablet_manager.cc:1428] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:19.760033 1188 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.761675 1188 tablet_bootstrap.cc:654] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.762796 1804 tablet_loader.cc:96] loaded metadata for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_0de0a6680c0f4531a75432a618576660 [id=72c128c9bf8849638e64f4eb99ec4c7e])
I20250110 02:51:19.763279 1804 tablet_loader.cc:96] loaded metadata for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9])
I20250110 02:51:19.763453 1184 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.763670 1184 ts_tablet_manager.cc:1397] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.008s
I20250110 02:51:19.764112 1804 tablet_loader.cc:96] loaded metadata for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad])
I20250110 02:51:19.764562 1804 tablet_loader.cc:96] loaded metadata for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_e7baf08453f44f56a442f1d355c8a32c [id=41e43455ee7c45a6960721c33a18b4d8])
I20250110 02:51:19.764936 1804 tablet_loader.cc:96] loaded metadata for tablet e2ab2376576448deb87f607b3e5a4f1c (table default.Table_ce235042d71840c0a05ae04e253c9517 [id=5ba8eda7053b45e2b27844e97e5c513a])
I20250110 02:51:19.765009 1184 raft_consensus.cc:357] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.765304 1184 raft_consensus.cc:383] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.765389 1184 raft_consensus.cc:738] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.765709 1804 tablet_loader.cc:96] loaded metadata for tablet e4726347afd649afa942d74df8c013fe (table default.Table_633f75bba62141e9a0075b23b2ec4bc0 [id=9b6434aff03e40a086c035ef1a731d3a])
I20250110 02:51:19.765777 1184 consensus_queue.cc:260] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.767927 1804 tablet_loader.cc:96] loaded metadata for tablet e55ee565b7f14936a2cb4cbed716c153 (table default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d])
I20250110 02:51:19.768620 1804 tablet_loader.cc:96] loaded metadata for tablet e804fe33c8ae489bbc4d3765c64b59bd (table default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d])
I20250110 02:51:19.768690 1825 raft_consensus.cc:491] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.768806 1825 raft_consensus.cc:513] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.769237 1804 tablet_loader.cc:96] loaded metadata for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7])
I20250110 02:51:19.769424 1825 leader_election.cc:290] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.769829 1804 tablet_loader.cc:96] loaded metadata for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150])
I20250110 02:51:19.770296 1804 tablet_loader.cc:96] loaded metadata for tablet eadb829d657346eabcb7d2aa34b33fe2 (table default.Table_93c93b5d73f24f75afbcb7660706db8a [id=5abe0f9c7bb9402c941464132d4ca4ae])
I20250110 02:51:19.770287 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.770761 1804 tablet_loader.cc:96] loaded metadata for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e])
W20250110 02:51:19.770972 709 leader_election.cc:343] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.771276 1804 tablet_loader.cc:96] loaded metadata for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819])
I20250110 02:51:19.774295 1184 ts_tablet_manager.cc:1428] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250110 02:51:19.771997 1804 tablet_loader.cc:96] loaded metadata for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09])
I20250110 02:51:19.780052 1680 consensus_queue.cc:1035] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.780325 1184 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.780653 1759 raft_consensus.cc:491] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.781780 1804 tablet_loader.cc:96] loaded metadata for tablet f3e808cfa6df40478892465f7bf5a8e5 (table default.Table_a171d415e02a4113afd448d388d8f124 [id=fa45e9ce8e684b3191fb25c7b33246ed])
I20250110 02:51:19.783018 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.783636 711 leader_election.cc:343] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.783741 711 leader_election.cc:304] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.784221 1825 raft_consensus.cc:2743] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.784257 1184 tablet_bootstrap.cc:654] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.786887 1804 tablet_loader.cc:96] loaded metadata for tablet f63d69d04b494a848653636c9e259105 (table default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310])
I20250110 02:51:19.787096 1188 tablet_bootstrap.cc:492] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.787238 1188 ts_tablet_manager.cc:1397] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20250110 02:51:19.787518 1804 tablet_loader.cc:96] loaded metadata for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584])
I20250110 02:51:19.787770 1825 raft_consensus.cc:491] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.787876 1825 raft_consensus.cc:513] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.788164 1804 tablet_loader.cc:96] loaded metadata for tablet f69d0c201c524b7e95c9d3851b8dd8c2 (table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6])
I20250110 02:51:19.788348 1825 leader_election.cc:290] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.789235 1804 tablet_loader.cc:96] loaded metadata for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9])
I20250110 02:51:19.789337 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.789875 1804 tablet_loader.cc:96] loaded metadata for tablet fb2272dfd0b14113afa3e13b38de6f8c (table default.Table_9ec7ac89c0a64225b6bdae4e65b0e8c4 [id=9643d639878f4122bf42aeca200f9c17])
W20250110 02:51:19.789942 709 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.790294 1804 tablet_loader.cc:96] loaded metadata for tablet fc1324eeac9446279304f442e2fad488 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c])
I20250110 02:51:19.790486 1188 raft_consensus.cc:357] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.790704 1188 raft_consensus.cc:383] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.790794 1188 raft_consensus.cc:738] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.791195 1825 raft_consensus.cc:491] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.791292 1825 raft_consensus.cc:513] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.791472 1804 tablet_loader.cc:96] loaded metadata for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d])
I20250110 02:51:19.792037 1825 leader_election.cc:290] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.792050 1804 tablet_loader.cc:96] loaded metadata for tablet ff57f894a5b54dee91452cbb4cb38b0c (table default.Table_c9d9d0b3b4444dc5b2050da6292dcf1e [id=6eaaca051e6c496eaef0567ca4b21030])
I20250110 02:51:19.792325 1726 raft_consensus.cc:491] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.792465 1726 raft_consensus.cc:513] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.793080 1726 leader_election.cc:290] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.793694 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:19.793828 1804 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Loading table and tablet metadata into memory: real 1.234s user 0.185s sys 0.048s
I20250110 02:51:19.793932 1804 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 02:51:19.793915 776 raft_consensus.cc:2388] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:19.798485 1184 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.798628 1184 ts_tablet_manager.cc:1397] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20250110 02:51:19.799291 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.799355 1184 raft_consensus.cc:357] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.799506 1184 raft_consensus.cc:383] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.799547 1184 raft_consensus.cc:738] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:19.800066 709 leader_election.cc:343] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.791014 1188 consensus_queue.cc:260] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.799932 1184 consensus_queue.cc:260] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.800632 1188 ts_tablet_manager.cc:1428] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250110 02:51:19.800817 1759 raft_consensus.cc:513] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.801476 1759 leader_election.cc:290] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:19.801599 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.801633 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec1dba54a247d5ba9f9e053aa3b645" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.801424 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.802167 1061 raft_consensus.cc:3054] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.802232 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.803262 853 leader_election.cc:304] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:19.803542 1184 ts_tablet_manager.cc:1428] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:19.803786 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0d22fd75fb474ebf07b624499a6f7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.803931 1184 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.804318 994 leader_election.cc:304] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:19.804756 711 leader_election.cc:343] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.804849 711 leader_election.cc:304] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.805073 1825 raft_consensus.cc:2743] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:19.805251 711 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.805333 711 leader_election.cc:304] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.805491 1825 raft_consensus.cc:2743] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.806681 1726 raft_consensus.cc:3054] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.808617 1184 tablet_bootstrap.cc:654] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.808974 1759 raft_consensus.cc:2743] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.809580 1726 raft_consensus.cc:2743] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.809819 1726 consensus_queue.cc:1035] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.812551 1188 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.812564 1804 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:19.812633 1804 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 02:51:19.814926 1181 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.815254 1188 tablet_bootstrap.cc:654] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.822185 1181 tablet_bootstrap.cc:654] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.828030 996 consensus_peers.cc:487] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46a4745356e3423f9d4892d242da1c48. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:19.829972 1184 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.832073 1184 ts_tablet_manager.cc:1397] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20250110 02:51:19.832813 1184 raft_consensus.cc:357] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.832986 1184 raft_consensus.cc:383] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.833048 1184 raft_consensus.cc:738] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.833266 1184 consensus_queue.cc:260] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.834434 1188 tablet_bootstrap.cc:492] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.834609 1188 ts_tablet_manager.cc:1397] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250110 02:51:19.835263 1804 catalog_manager.cc:1505] Loading token signing keys...
I20250110 02:51:19.835376 1188 raft_consensus.cc:357] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.835582 1188 raft_consensus.cc:383] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.835651 1188 raft_consensus.cc:738] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.835877 1188 consensus_queue.cc:260] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.836330 1188 ts_tablet_manager.cc:1428] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:19.842077 1184 ts_tablet_manager.cc:1428] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:19.846567 1916 raft_consensus.cc:491] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.846779 1184 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.846729 1916 raft_consensus.cc:513] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.847411 1916 leader_election.cc:290] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.849511 1184 tablet_bootstrap.cc:654] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.849686 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.849865 776 raft_consensus.cc:2463] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.850291 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.850502 1059 raft_consensus.cc:2463] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.850869 853 leader_election.cc:304] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.851204 1726 consensus_queue.cc:1035] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.851580 1726 raft_consensus.cc:2798] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.851673 1726 raft_consensus.cc:491] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.851740 1726 raft_consensus.cc:3054] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.856043 1804 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Loaded TSK: 0
I20250110 02:51:19.854243 1726 raft_consensus.cc:513] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.857426 1726 leader_election.cc:290] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.858047 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:19.858207 776 raft_consensus.cc:3054] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.858897 1804 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 02:51:19.860872 776 raft_consensus.cc:2463] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.861610 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0d4d6e02b1d4334a9993d320217f1f8" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:19.861817 1058 raft_consensus.cc:3054] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.865178 1058 raft_consensus.cc:2463] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:19.869170 1726 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.869424 1726 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.862093 1188 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.862501 853 leader_election.cc:304] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:19.868978 1770 consensus_queue.cc:1035] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.870645 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:19.870823 774 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.870950 1184 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.871066 1184 ts_tablet_manager.cc:1397] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20250110 02:51:19.871366 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.871549 1058 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:19.872236 1184 raft_consensus.cc:357] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.872408 1184 raft_consensus.cc:383] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.872453 1184 raft_consensus.cc:738] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.872741 1184 consensus_queue.cc:260] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.873571 1184 ts_tablet_manager.cc:1428] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.873813 1184 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.875674 1184 tablet_bootstrap.cc:654] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.877985 1916 raft_consensus.cc:2798] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:19.878196 853 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:19.879137 1831 raft_consensus.cc:2798] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:19.879627 1825 raft_consensus.cc:491] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.879763 1825 raft_consensus.cc:513] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.880323 1825 leader_election.cc:290] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.881078 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.881600 711 leader_election.cc:343] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.882941 1188 tablet_bootstrap.cc:654] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.886459 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fb805b3e537466ab7fe0185421ad5aa from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:19.889153 1443 master_service.cc:946] ReplaceTablet: received request to replace tablet a0ddab9e847445999ec97eac4352d9c2 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:19.889623 1184 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.889706 1184 ts_tablet_manager.cc:1397] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20250110 02:51:19.890249 1184 raft_consensus.cc:357] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.890362 1184 raft_consensus.cc:383] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.890471 1184 raft_consensus.cc:738] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.890661 1184 consensus_queue.cc:260] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.891263 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet d4f1d916a5ea48708c2c1a61c92fe137 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:19.891395 1184 ts_tablet_manager.cc:1428] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:19.891638 1184 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.893304 1184 tablet_bootstrap.cc:654] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.888391 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_bd274cbe850641b69ba5a4b6fa3f2474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:19.892196 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_4ef1477d7e314ddbaa6fd875538bc552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:19.888979 1448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_09dd77d568b8468cbad2375415ecb197" } new_table_name: "default.Table_74173cc3c3b8435f897e1710a0711981" modify_external_catalogs: true
W20250110 02:51:19.895901 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd274cbe850641b69ba5a4b6fa3f2474 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:19.895910 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ef1477d7e314ddbaa6fd875538bc552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:19.890000 1441 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e7baf08453f44f56a442f1d355c8a32c" } new_table_name: "default.Table_aee0556087634aedb308add0c4725dae" modify_external_catalogs: true
I20250110 02:51:19.888715 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7ac37bfbe9d7409cbd07212eceafa582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:19.896699 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ac37bfbe9d7409cbd07212eceafa582 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:19.890798 1439 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_9b414f28d5514d17a68f7f198a2054d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:19.897435 1439 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b414f28d5514d17a68f7f198a2054d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:19.890406 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_0943d6c040ee49e1b0ccdc1ca19457af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:19.898978 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0943d6c040ee49e1b0ccdc1ca19457af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:19.884721 1444 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_38cc13d405674486842b52efad1baed0" } new_table_name: "default.Table_bbc18e2b41314211bf9690b0b60739a1" modify_external_catalogs: true
I20250110 02:51:19.904608 1181 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:19.904769 1181 ts_tablet_manager.cc:1397] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.165s user 0.006s sys 0.002s
I20250110 02:51:19.905558 1181 raft_consensus.cc:357] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.905791 1181 raft_consensus.cc:383] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.905886 1181 raft_consensus.cc:738] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.907094 1184 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:19.907212 1184 ts_tablet_manager.cc:1397] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20250110 02:51:19.884791 1726 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.884884 1916 raft_consensus.cc:695] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:19.890435 1831 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:19.909252 1831 raft_consensus.cc:3054] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.910607 1184 raft_consensus.cc:357] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.910764 1184 raft_consensus.cc:383] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.910815 1184 raft_consensus.cc:738] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.910974 1184 consensus_queue.cc:260] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.911500 1184 ts_tablet_manager.cc:1428] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:19.911728 1184 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:19.912038 1831 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.890755 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.891009 853 consensus_peers.cc:487] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:19.913012 1184 tablet_bootstrap.cc:654] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:19.908934 1916 consensus_queue.cc:237] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.913611 915 raft_consensus.cc:3054] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.913820 1181 consensus_queue.cc:260] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.914487 1181 ts_tablet_manager.cc:1428] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:19.914743 1181 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:19.916178 1181 tablet_bootstrap.cc:654] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.916613 709 leader_election.cc:343] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.916738 709 leader_election.cc:304] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.917299 1825 raft_consensus.cc:2743] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.917469 1831 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.919648 1770 consensus_queue.cc:1035] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.937767 1825 raft_consensus.cc:491] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.937916 1825 raft_consensus.cc:513] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:19.938630 1825 leader_election.cc:290] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.939127 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:19.939555 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:19.939713 711 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.940286 709 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.940429 709 leader_election.cc:304] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.940951 1825 raft_consensus.cc:2743] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.945470 917 raft_consensus.cc:3054] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:19.949836 1759 consensus_queue.cc:1035] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.957989 1825 raft_consensus.cc:491] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.958145 1825 raft_consensus.cc:513] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.958815 1825 leader_election.cc:290] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:19.959365 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28340f0f9a2043a199c74f13755eb702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:19.959408 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28340f0f9a2043a199c74f13755eb702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:19.959985 711 leader_election.cc:343] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:19.960114 709 leader_election.cc:343] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:19.960255 709 leader_election.cc:304] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:19.960789 1825 raft_consensus.cc:2743] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:19.971006 1759 consensus_queue.cc:1035] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:19.976820 1916 raft_consensus.cc:491] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:19.977036 1916 raft_consensus.cc:513] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.977898 1916 leader_election.cc:290] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:19.918040 1188 tablet_bootstrap.cc:492] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:19.979460 1188 ts_tablet_manager.cc:1397] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.118s user 0.004s sys 0.000s
I20250110 02:51:19.980530 1188 raft_consensus.cc:357] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.980720 1188 raft_consensus.cc:383] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:19.980808 1188 raft_consensus.cc:738] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:19.981051 1188 consensus_queue.cc:260] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:19.982331 1188 ts_tablet_manager.cc:1428] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:19.983300 1188 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:19.985117 1188 tablet_bootstrap.cc:654] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:19.993156 996 consensus_peers.cc:487] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e744cdc1fcb94d46b48b684763ecf1bf. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.006584 1759 raft_consensus.cc:491] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.006841 1759 raft_consensus.cc:513] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.007699 1759 leader_election.cc:290] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.008910 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.008921 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.009297 775 raft_consensus.cc:2463] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:20.009557 994 leader_election.cc:343] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.009778 996 leader_election.cc:304] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:20.010365 1759 raft_consensus.cc:2798] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.010460 1759 raft_consensus.cc:491] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.010540 1759 raft_consensus.cc:3054] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.010828 1849 consensus_queue.cc:1035] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.013826 1759 raft_consensus.cc:513] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.015190 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:20.015625 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7902775b8db247a89d9b20d0ee880e36" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.015802 774 raft_consensus.cc:3054] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:20.015870 994 leader_election.cc:343] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.016265 1825 raft_consensus.cc:491] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.016371 1825 raft_consensus.cc:513] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.016913 1825 leader_election.cc:290] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.017570 1759 leader_election.cc:290] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.018738 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.019471 711 leader_election.cc:343] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.019685 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.020195 709 leader_election.cc:343] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.020222 774 raft_consensus.cc:2463] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.020285 709 leader_election.cc:304] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.020591 1825 raft_consensus.cc:2743] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.020644 996 leader_election.cc:304] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:20.021278 1759 raft_consensus.cc:2798] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.021440 1759 raft_consensus.cc:695] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:20.022267 1759 consensus_queue.cc:237] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.027665 1680 consensus_queue.cc:1035] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:20.033696 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 120248us
I20250110 02:51:20.035207 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.035205 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.035383 1059 raft_consensus.cc:3054] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.035414 774 raft_consensus.cc:3054] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.038125 1059 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.038254 774 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.039304 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf711522866d46439d7ee69aa63d7f39" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.039491 1061 raft_consensus.cc:2463] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.042809 853 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.043270 853 leader_election.cc:304] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.042980 1832 raft_consensus.cc:491] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.043718 1832 raft_consensus.cc:513] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.043884 1825 raft_consensus.cc:491] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.044054 1825 raft_consensus.cc:513] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.044390 1832 leader_election.cc:290] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.043468 1953 raft_consensus.cc:2798] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.044647 1726 raft_consensus.cc:2798] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.044723 1825 leader_election.cc:290] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.044744 1726 raft_consensus.cc:491] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.044822 1726 raft_consensus.cc:3054] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.045130 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.045233 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.045547 711 leader_election.cc:343] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.045634 709 leader_election.cc:343] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.045689 711 leader_election.cc:304] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.046002 1825 raft_consensus.cc:2743] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.047382 1726 raft_consensus.cc:513] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.047830 1184 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.047955 1184 ts_tablet_manager.cc:1397] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.136s user 0.022s sys 0.040s
I20250110 02:51:20.048636 1726 leader_election.cc:290] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.048666 1184 raft_consensus.cc:357] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.048833 1184 raft_consensus.cc:383] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.048906 1184 raft_consensus.cc:738] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.049125 1953 raft_consensus.cc:695] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.049116 1184 consensus_queue.cc:260] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.050357 1953 consensus_queue.cc:237] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:20.051400 853 leader_election.cc:343] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:20.052974 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.053136 1061 raft_consensus.cc:2463] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.053227 1184 ts_tablet_manager.cc:1428] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:20.053506 1184 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.053651 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf711522866d46439d7ee69aa63d7f39" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.053822 1061 raft_consensus.cc:3054] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:20.054841 853 leader_election.cc:343] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:20.055037 853 leader_election.cc:304] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.055226 1184 tablet_bootstrap.cc:654] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.055289 1726 raft_consensus.cc:2798] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.055359 1726 raft_consensus.cc:491] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.055459 1726 raft_consensus.cc:3054] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.056553 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.056701 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 3fb805b3e537466ab7fe0185421ad5aa
I20250110 02:51:20.056764 775 raft_consensus.cc:2463] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.058022 1726 raft_consensus.cc:513] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.058817 1726 leader_election.cc:290] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.059367 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.059561 775 raft_consensus.cc:3054] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.063006 775 raft_consensus.cc:2463] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.063161 1450 catalog_manager.cc:6596] ReplaceTablet: tablet 3fb805b3e537466ab7fe0185421ad5aa deleted and replaced by tablet d48a22b9642a442e83a3ecb7117d5c80
I20250110 02:51:20.063536 853 leader_election.cc:304] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:20.063869 1726 raft_consensus.cc:2798] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.064079 1726 raft_consensus.cc:695] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.065488 1726 consensus_queue.cc:237] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.068903 1188 tablet_bootstrap.cc:492] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.069165 1188 ts_tablet_manager.cc:1397] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20250110 02:51:20.070190 1188 raft_consensus.cc:357] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.078933 1188 raft_consensus.cc:383] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.079030 1188 raft_consensus.cc:738] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.079257 1188 consensus_queue.cc:260] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.079916 1188 ts_tablet_manager.cc:1428] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:20.080214 1188 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.082075 1188 tablet_bootstrap.cc:654] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.085739 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:20.087102 1184 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.087231 1184 ts_tablet_manager.cc:1397] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250110 02:51:20.087986 1184 raft_consensus.cc:357] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.088207 1184 raft_consensus.cc:383] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.088213 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:20.088260 1184 raft_consensus.cc:738] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.088441 1184 consensus_queue.cc:260] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.089066 1184 ts_tablet_manager.cc:1428] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:20.089757 1680 consensus_queue.cc:1035] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250110 02:51:20.090996 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558a2c03f6b42a6befdc48e3af23a6d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.091447 1061 raft_consensus.cc:2463] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.095239 1059 raft_consensus.cc:3054] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.097721 1184 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.098145 1059 raft_consensus.cc:2463] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:20.098409 996 consensus_peers.cc:487] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 585320c0970c4097b195caaae33789b5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.100054 1184 tablet_bootstrap.cc:654] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:20.102222 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:20.107884 1184 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.108029 1184 ts_tablet_manager.cc:1397] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250110 02:51:20.108829 1184 raft_consensus.cc:357] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.109033 1184 raft_consensus.cc:383] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.109113 1184 raft_consensus.cc:738] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.109218 853 leader_election.cc:304] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.109318 1184 consensus_queue.cc:260] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.109972 1184 ts_tablet_manager.cc:1428] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:20.110319 1184 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.111410 1832 raft_consensus.cc:2798] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.111534 1832 raft_consensus.cc:695] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.111766 1832 consensus_queue.cc:237] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.114516 1181 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.114675 1181 ts_tablet_manager.cc:1397] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.200s user 0.003s sys 0.023s
I20250110 02:51:20.115504 1181 raft_consensus.cc:357] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.115756 1181 raft_consensus.cc:383] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.115851 1181 raft_consensus.cc:738] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.116144 1181 consensus_queue.cc:260] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.121409 1770 raft_consensus.cc:491] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.121438 1832 raft_consensus.cc:491] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.121579 1770 raft_consensus.cc:513] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.121627 1832 raft_consensus.cc:513] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.122397 1832 leader_election.cc:290] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:20.125216 853 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:20.125862 1184 tablet_bootstrap.cc:654] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.127722 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:20.128134 1188 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.128314 1188 ts_tablet_manager.cc:1397] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.002s
W20250110 02:51:20.129865 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:20.132510 1188 raft_consensus.cc:357] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.134874 1825 raft_consensus.cc:491] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.135149 1184 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.135026 1825 raft_consensus.cc:513] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.135270 1184 ts_tablet_manager.cc:1397] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.005s
I20250110 02:51:20.135675 1825 leader_election.cc:290] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.136209 1184 raft_consensus.cc:357] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.136441 1184 raft_consensus.cc:383] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.136487 1184 raft_consensus.cc:738] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.136711 1184 consensus_queue.cc:260] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.137355 1184 ts_tablet_manager.cc:1428] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:20.137701 1184 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.141003 1184 tablet_bootstrap.cc:654] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.143126 1188 raft_consensus.cc:383] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.143252 1188 raft_consensus.cc:738] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.143518 1188 consensus_queue.cc:260] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.144173 1188 ts_tablet_manager.cc:1428] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250110 02:51:20.144449 1188 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.146309 1188 tablet_bootstrap.cc:654] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.136476 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.147343 709 leader_election.cc:343] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.136958 1953 raft_consensus.cc:491] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.151901 1953 raft_consensus.cc:513] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.155582 1953 leader_election.cc:290] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.155932 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb5412b78104d8085878c58b191ab27" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.156173 916 raft_consensus.cc:2463] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:20.157116 853 leader_election.cc:343] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:20.158071 1188 tablet_bootstrap.cc:492] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.158281 1188 ts_tablet_manager.cc:1397] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250110 02:51:20.158527 1825 raft_consensus.cc:491] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.158763 1825 raft_consensus.cc:513] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.159296 1188 raft_consensus.cc:357] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.159454 1825 leader_election.cc:290] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.159524 1188 raft_consensus.cc:383] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.159584 1188 raft_consensus.cc:738] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.159934 1770 leader_election.cc:290] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.160297 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.160753 709 leader_election.cc:343] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.160948 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66603a0ee8c4082807c5a6aabe7e339" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.161135 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.161777 994 leader_election.cc:304] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.162012 1188 consensus_queue.cc:260] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.161160 1059 raft_consensus.cc:2463] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
W20250110 02:51:20.163935 711 leader_election.cc:343] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.163762 853 leader_election.cc:304] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.164067 711 leader_election.cc:304] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.164345 1953 raft_consensus.cc:2798] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.164448 1953 raft_consensus.cc:491] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.164583 1953 raft_consensus.cc:3054] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.164670 1825 raft_consensus.cc:2743] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.167253 1953 raft_consensus.cc:513] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.168444 1953 leader_election.cc:290] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.163182 1188 ts_tablet_manager.cc:1428] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20250110 02:51:20.168826 853 leader_election.cc:343] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:20.169157 1188 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.170755 1188 tablet_bootstrap.cc:654] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.174768 1770 raft_consensus.cc:2798] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250110 02:51:20.175777 996 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 1fb5412b78104d8085878c58b191ab27
I20250110 02:51:20.176517 1770 raft_consensus.cc:491] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.176645 1770 raft_consensus.cc:3054] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.180519 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.180595 853 consensus_peers.cc:487] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.180593 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66603a0ee8c4082807c5a6aabe7e339" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.180809 1181 ts_tablet_manager.cc:1428] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250110 02:51:20.181741 994 consensus_peers.cc:487] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.181797 1060 raft_consensus.cc:3054] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:20.181876 711 leader_election.cc:343] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.181977 711 leader_election.cc:304] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.181031 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ace37b96844328ccdb821dad3c5ea" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.185034 1188 tablet_bootstrap.cc:492] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.184973 1770 raft_consensus.cc:513] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.185186 1188 ts_tablet_manager.cc:1397] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250110 02:51:20.186136 1188 raft_consensus.cc:357] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.186745 1059 raft_consensus.cc:2463] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.187217 853 leader_election.cc:304] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.187690 1832 raft_consensus.cc:2798] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.187774 1832 raft_consensus.cc:491] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.187872 1832 raft_consensus.cc:3054] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.190253 1188 raft_consensus.cc:383] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.190404 1188 raft_consensus.cc:738] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.190927 1770 leader_election.cc:290] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.190959 1825 raft_consensus.cc:2743] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.191118 1825 raft_consensus.cc:491] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.191241 1825 raft_consensus.cc:513] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.191594 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb5412b78104d8085878c58b191ab27" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:20.191781 916 raft_consensus.cc:3054] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.191787 1825 leader_election.cc:290] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.192549 1060 raft_consensus.cc:2463] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:20.192889 996 leader_election.cc:343] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 1fb5412b78104d8085878c58b191ab27
I20250110 02:51:20.190326 1832 raft_consensus.cc:513] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.193105 853 leader_election.cc:304] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.193521 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.193624 1832 leader_election.cc:290] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.193732 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:20.193773 1832 raft_consensus.cc:2798] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.194007 1832 raft_consensus.cc:695] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.194294 916 raft_consensus.cc:2463] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.194264 1832 consensus_queue.cc:237] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.194579 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.194968 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:20.195025 709 leader_election.cc:343] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.195318 711 leader_election.cc:343] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.195343 1181 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.194976 994 leader_election.cc:304] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.195401 711 leader_election.cc:304] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.195691 1825 raft_consensus.cc:2743] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.196336 1770 raft_consensus.cc:2798] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.196498 1770 raft_consensus.cc:695] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
W20250110 02:51:20.196527 853 leader_election.cc:343] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:20.196933 1770 consensus_queue.cc:237] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.197185 1181 tablet_bootstrap.cc:654] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.198062 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10ace37b96844328ccdb821dad3c5ea" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.198242 1060 raft_consensus.cc:3054] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.201174 1060 raft_consensus.cc:2463] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.201242 1953 raft_consensus.cc:491] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.201398 1953 raft_consensus.cc:513] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.201539 853 leader_election.cc:304] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.201922 1957 raft_consensus.cc:2798] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.202075 1957 raft_consensus.cc:695] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.202400 1957 consensus_queue.cc:237] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.203301 1953 leader_election.cc:290] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.206530 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.206795 1060 raft_consensus.cc:2463] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.207306 853 leader_election.cc:304] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.207680 1953 raft_consensus.cc:2798] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.207772 1825 raft_consensus.cc:491] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.207821 1953 raft_consensus.cc:491] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.207924 1953 raft_consensus.cc:3054] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.207911 1825 raft_consensus.cc:513] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.208571 1825 leader_election.cc:290] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.209347 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.209475 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.209820 709 leader_election.cc:343] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.210054 711 leader_election.cc:343] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.210186 711 leader_election.cc:304] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.210543 1825 raft_consensus.cc:2743] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.210708 1188 consensus_queue.cc:260] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.211423 1953 raft_consensus.cc:513] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.212424 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.212635 1059 raft_consensus.cc:3054] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.213024 1188 ts_tablet_manager.cc:1428] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
W20250110 02:51:20.213316 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:20.213538 1995 tablet_replica.cc:331] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:20.213757 1995 raft_consensus.cc:2238] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:20.213989 1995 raft_consensus.cc:2267] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:20.214771 1188 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.214895 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.215389 1995 ts_tablet_manager.cc:1905] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:20.215378 775 raft_consensus.cc:2463] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.215579 1059 raft_consensus.cc:2463] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.216023 853 leader_election.cc:304] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.216292 1726 raft_consensus.cc:2798] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.216432 1726 raft_consensus.cc:695] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.216653 1726 consensus_queue.cc:237] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.218514 1953 leader_election.cc:290] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.218792 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6636a3b7f2c4e2fb0ef76f9ab25dcfb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.219017 775 raft_consensus.cc:3054] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.220070 1849 raft_consensus.cc:491] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.220204 1849 raft_consensus.cc:513] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.221355 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127de45264f4660b843734a530eb310" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.221589 915 raft_consensus.cc:2463] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:20.222033 994 leader_election.cc:304] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.222400 1770 raft_consensus.cc:2798] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.222561 1770 raft_consensus.cc:491] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.222702 1770 raft_consensus.cc:3054] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:20.222868 996 leader_election.cc:343] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0127de45264f4660b843734a530eb310
I20250110 02:51:20.223369 1849 leader_election.cc:290] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.225692 1770 raft_consensus.cc:513] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.226838 1995 ts_tablet_manager.cc:1918] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:20.227715 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127de45264f4660b843734a530eb310" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:20.227946 915 raft_consensus.cc:3054] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.231127 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskadt4RW/build/asan/bin/kudu
/tmp/dist-test-taskadt4RW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:44449
--webserver_interface=127.0.104.61
--webserver_port=46707
--builtin_ntp_servers=127.0.104.20:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 02:51:20.231904 1825 raft_consensus.cc:491] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.232017 1825 raft_consensus.cc:513] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.232518 1825 leader_election.cc:290] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.232983 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.233412 709 leader_election.cc:343] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.233745 1188 tablet_bootstrap.cc:654] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.235620 915 raft_consensus.cc:2463] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.242128 1825 raft_consensus.cc:491] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.242244 1825 raft_consensus.cc:513] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.242769 1825 leader_election.cc:290] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.243265 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.243996 709 leader_election.cc:343] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.228680 996 leader_election.cc:343] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 0127de45264f4660b843734a530eb310
I20250110 02:51:20.244449 994 leader_election.cc:304] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.229004 1770 leader_election.cc:290] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.244704 1849 raft_consensus.cc:2798] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.244853 1849 raft_consensus.cc:695] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:20.245097 1849 consensus_queue.cc:237] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.230166 1181 tablet_bootstrap.cc:492] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.245998 1181 ts_tablet_manager.cc:1397] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.005s
I20250110 02:51:20.246021 1995 log.cc:1198] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f14f06ba7d0f44418693e36f6bc6d888
I20250110 02:51:20.246441 1995 ts_tablet_manager.cc:1939] T f14f06ba7d0f44418693e36f6bc6d888 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:20.246518 1188 tablet_bootstrap.cc:492] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.246666 1188 ts_tablet_manager.cc:1397] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20250110 02:51:20.233922 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.247213 1825 raft_consensus.cc:491] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.243657 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.247393 1825 raft_consensus.cc:513] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.247442 1188 raft_consensus.cc:357] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.247658 1188 raft_consensus.cc:383] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.247777 1188 raft_consensus.cc:738] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.248132 1181 raft_consensus.cc:357] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.248245 1825 leader_election.cc:290] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.248103 1188 consensus_queue.cc:260] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.248323 1181 raft_consensus.cc:383] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.248418 1181 raft_consensus.cc:738] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.248647 1181 consensus_queue.cc:260] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.248816 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.249164 1181 ts_tablet_manager.cc:1428] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250110 02:51:20.249251 709 leader_election.cc:343] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.249445 1181 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.249569 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f14f06ba7d0f44418693e36f6bc6d888 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
I20250110 02:51:20.249832 1188 ts_tablet_manager.cc:1428] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:20.251345 915 raft_consensus.cc:3054] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.251346 1188 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.252187 1181 tablet_bootstrap.cc:654] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.252981 775 raft_consensus.cc:2463] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.253175 1188 tablet_bootstrap.cc:654] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.254297 1989 raft_consensus.cc:491] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.254683 1989 raft_consensus.cc:513] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.255267 1989 leader_election.cc:290] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.255816 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.256278 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.257000 711 leader_election.cc:343] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.257097 711 leader_election.cc:304] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:20.257078 994 leader_election.cc:343] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.257447 711 leader_election.cc:343] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.258406 1181 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.258448 711 leader_election.cc:304] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.258689 1181 ts_tablet_manager.cc:1397] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250110 02:51:20.258991 1825 raft_consensus.cc:2743] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.257643 1991 raft_consensus.cc:2743] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.257843 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.259203 775 raft_consensus.cc:2463] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:20.259429 711 leader_election.cc:343] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.259621 1181 raft_consensus.cc:357] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.259944 1181 raft_consensus.cc:383] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.259858 711 leader_election.cc:304] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.260048 1181 raft_consensus.cc:738] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.260252 1991 raft_consensus.cc:2743] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.260484 1184 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.260361 1181 consensus_queue.cc:260] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.260536 996 leader_election.cc:304] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:20.260668 1184 ts_tablet_manager.cc:1397] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.123s user 0.018s sys 0.010s
I20250110 02:51:20.261056 1181 ts_tablet_manager.cc:1428] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:20.261102 1770 raft_consensus.cc:2798] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.261205 1770 raft_consensus.cc:491] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.261308 1770 raft_consensus.cc:3054] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.261449 1181 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.261529 1184 raft_consensus.cc:357] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.261682 1184 raft_consensus.cc:383] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.261729 1184 raft_consensus.cc:738] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.261920 1184 consensus_queue.cc:260] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.264030 1770 raft_consensus.cc:513] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.264168 1181 tablet_bootstrap.cc:654] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.265079 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:20.265515 994 leader_election.cc:343] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.266479 1770 leader_election.cc:290] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.266836 1188 tablet_bootstrap.cc:492] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.266988 1188 ts_tablet_manager.cc:1397] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250110 02:51:20.267838 1188 raft_consensus.cc:357] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.268079 1188 raft_consensus.cc:383] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.268177 1188 raft_consensus.cc:738] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.268394 1188 consensus_queue.cc:260] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.268687 1443 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet a0ddab9e847445999ec97eac4352d9c2
I20250110 02:51:20.269248 1443 catalog_manager.cc:6596] ReplaceTablet: tablet a0ddab9e847445999ec97eac4352d9c2 deleted and replaced by tablet 81e26341525d462e9042b1ab06cf7ac3
I20250110 02:51:20.269933 1188 ts_tablet_manager.cc:1428] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:20.270442 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:20.270556 1188 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:20.270931 1441 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e27f7ffc50494854ba34e1fbce9bee1b consensus state has no leader
W20250110 02:51:20.271310 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:20.271764 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:20.273638 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:20.274960 1181 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.275112 1181 ts_tablet_manager.cc:1397] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250110 02:51:20.276058 1181 raft_consensus.cc:357] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.276319 1181 raft_consensus.cc:383] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.276427 1181 raft_consensus.cc:738] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.276839 1181 consensus_queue.cc:260] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.277565 1181 ts_tablet_manager.cc:1428] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:20.278312 1181 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:20.280103 1444 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0059fbcbfb7a4f239a5947f702d21c0f consensus state has no leader
I20250110 02:51:20.280364 1444 master_service.cc:946] ReplaceTablet: received request to replace tablet d4f1d916a5ea48708c2c1a61c92fe137 from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:20.280884 1448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab302fedec4f4c03989b5d3932dc8d4c consensus state has no leader
I20250110 02:51:20.281738 1181 tablet_bootstrap.cc:654] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.282912 1188 tablet_bootstrap.cc:654] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.284484 1181 tablet_bootstrap.cc:492] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.284696 1181 ts_tablet_manager.cc:1397] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250110 02:51:20.285518 1181 raft_consensus.cc:357] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.285797 1181 raft_consensus.cc:383] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.285898 1181 raft_consensus.cc:738] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.286191 1181 consensus_queue.cc:260] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.286832 1441 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_618cae6951b649328aab854b7efa2a14" } new_table_name: "default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce" modify_external_catalogs: true
I20250110 02:51:20.287308 1181 ts_tablet_manager.cc:1428] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:20.287695 1181 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.288069 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_aad85d64a3ff4f87b6c4f767f39de7df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.288645 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aad85d64a3ff4f87b6c4f767f39de7df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.289022 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_bdaba3ed414148628957dbf5e4462911"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.289593 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdaba3ed414148628957dbf5e4462911 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.290817 1439 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_af9287df67924374975e9318755998f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.291317 1439 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af9287df67924374975e9318755998f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.291520 1457 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_055d44655c2743f3907923f2f49a33af" } new_table_name: "default.Table_52a56e3c3c3a47f0a95b71d69acc52b3" modify_external_catalogs: true
I20250110 02:51:20.292812 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_829a8c82f1a44c3b9b1b67047764be79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.293293 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_829a8c82f1a44c3b9b1b67047764be79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.296193 1444 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_de1fbb2884e2436193565eff3bc88357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.296672 1444 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de1fbb2884e2436193565eff3bc88357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.297643 1989 consensus_queue.cc:1035] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.300115 1181 tablet_bootstrap.cc:654] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.303084 755 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:20.304381 1450 catalog_manager.cc:5526] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:20.304977 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
W20250110 02:51:20.305464 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:20.305508 1989 consensus_queue.cc:1035] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:20.305693 996 consensus_peers.cc:487] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.306165 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.306327 775 raft_consensus.cc:3054] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.308455 1989 consensus_queue.cc:1035] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.309491 1456 catalog_manager.cc:5526] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:20.310104 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09]). Expected version 3 got 0
I20250110 02:51:20.310127 775 raft_consensus.cc:2463] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.310753 996 leader_election.cc:304] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:20.311158 1989 raft_consensus.cc:2798] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.311306 1989 raft_consensus.cc:695] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:20.311558 1989 consensus_queue.cc:237] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.311820 1990 raft_consensus.cc:491] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.311970 1990 raft_consensus.cc:513] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.312767 1990 leader_election.cc:290] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.313462 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.314113 994 leader_election.cc:343] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.319212 1991 raft_consensus.cc:491] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.319386 1991 raft_consensus.cc:513] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.320209 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.320397 774 raft_consensus.cc:2463] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:20.320381 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.320851 709 leader_election.cc:343] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.321184 1991 leader_election.cc:290] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.321928 996 leader_election.cc:304] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:20.322611 1989 raft_consensus.cc:2798] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.323834 1989 raft_consensus.cc:491] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.323927 1989 raft_consensus.cc:3054] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.324604 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.325502 1770 consensus_queue.cc:1035] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:20.326421 711 leader_election.cc:343] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.326663 711 leader_election.cc:304] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.327042 1991 raft_consensus.cc:2743] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.327334 1989 raft_consensus.cc:513] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.328522 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:20.328606 1989 leader_election.cc:290] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:20.329051 994 leader_election.cc:343] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.329734 1184 ts_tablet_manager.cc:1428] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.069s user 0.001s sys 0.001s
W20250110 02:51:20.330286 1434 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e27f7ffc50494854ba34e1fbce9bee1b consensus state has no leader
I20250110 02:51:20.330328 1184 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.335016 1188 tablet_bootstrap.cc:492] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.335283 1188 ts_tablet_manager.cc:1397] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.018s
I20250110 02:51:20.335819 1989 raft_consensus.cc:491] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.336076 1989 raft_consensus.cc:513] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.336753 1188 raft_consensus.cc:357] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.337278 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af3d61366b2432db53b18f2b7bb49d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.337535 916 raft_consensus.cc:2463] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:20.337361 1188 raft_consensus.cc:383] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.337762 1188 raft_consensus.cc:738] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.337985 994 leader_election.cc:304] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.338274 1188 consensus_queue.cc:260] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.340262 1188 ts_tablet_manager.cc:1428] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250110 02:51:20.341835 1188 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.343820 1188 tablet_bootstrap.cc:654] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.354458 1991 raft_consensus.cc:491] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.354646 1991 raft_consensus.cc:513] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.354733 1770 raft_consensus.cc:2798] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.354854 1770 raft_consensus.cc:491] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.355506 1991 leader_election.cc:290] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.356418 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.357226 709 leader_election.cc:343] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.365201 1184 tablet_bootstrap.cc:654] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.367394 1991 raft_consensus.cc:491] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.367602 1991 raft_consensus.cc:513] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.368402 1991 leader_election.cc:290] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.369228 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f15078fc22f4b32a48c7190b36e2192" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.369956 709 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.370342 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69d0c201c524b7e95c9d3851b8dd8c2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.370663 776 raft_consensus.cc:3054] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.373551 776 raft_consensus.cc:2463] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.381078 1184 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.381922 1184 ts_tablet_manager.cc:1397] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20250110 02:51:20.382699 1184 raft_consensus.cc:357] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.382944 1184 raft_consensus.cc:383] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.383010 1184 raft_consensus.cc:738] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.383210 1184 consensus_queue.cc:260] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.383682 1184 ts_tablet_manager.cc:1428] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:20.384006 1184 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.385833 1184 tablet_bootstrap.cc:654] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.388625 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4f88a7002f413fbd63491fe8e1fe6d (DEFAULT_TABLE table=default.Table_2009672da4c945ec8fbbac6f5fe0988a [id=c409e93003184d2282188bd417dd3580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.389036 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4f88a7002f413fbd63491fe8e1fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.390213 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4f88a7002f413fbd63491fe8e1fe6d (DEFAULT_TABLE table=default.Table_2009672da4c945ec8fbbac6f5fe0988a [id=c409e93003184d2282188bd417dd3580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.390589 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4f88a7002f413fbd63491fe8e1fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.394703 1991 raft_consensus.cc:491] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.394837 1991 raft_consensus.cc:513] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.395382 1991 leader_election.cc:290] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.395989 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.396533 709 leader_election.cc:343] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.399868 894 tablet_service.cc:1467] Processing CreateTablet for tablet 9da4d8583ce74605a80da29ac030692a (DEFAULT_TABLE table=default.Table_34ce8eea83364d1ca7d8cfa100115b45 [id=4696b770f7514e2f9d5aab8e16ad9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.400207 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da4d8583ce74605a80da29ac030692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.402004 891 tablet_service.cc:1467] Processing CreateTablet for tablet b4f1782c46a44ed091f4c533226c71f2 (DEFAULT_TABLE table=default.Table_18eb7bac666944ddaf84da77edd4462b [id=5d3953c49f6f46b29e8ea2cd1f72f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.402833 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f1782c46a44ed091f4c533226c71f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.403280 1443 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_3e713a5d93834ea3be890e774e266d83" } modify_external_catalogs: true
I20250110 02:51:20.403476 1443 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_3e713a5d93834ea3be890e774e266d83" } modify_external_catalogs: true
I20250110 02:51:20.403584 743 tablet_service.cc:1467] Processing CreateTablet for tablet b4f1782c46a44ed091f4c533226c71f2 (DEFAULT_TABLE table=default.Table_18eb7bac666944ddaf84da77edd4462b [id=5d3953c49f6f46b29e8ea2cd1f72f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.404053 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f1782c46a44ed091f4c533226c71f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.404007 892 tablet_service.cc:1467] Processing CreateTablet for tablet f3e808cfa6df40478892465f7bf5a8e5 (DEFAULT_TABLE table=default.Table_a171d415e02a4113afd448d388d8f124 [id=fa45e9ce8e684b3191fb25c7b33246ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.404331 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e808cfa6df40478892465f7bf5a8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.405364 751 tablet_service.cc:1467] Processing CreateTablet for tablet f3e808cfa6df40478892465f7bf5a8e5 (DEFAULT_TABLE table=default.Table_a171d415e02a4113afd448d388d8f124 [id=fa45e9ce8e684b3191fb25c7b33246ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.405758 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e808cfa6df40478892465f7bf5a8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.407203 1188 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.407361 1188 ts_tablet_manager.cc:1397] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
I20250110 02:51:20.408141 1188 raft_consensus.cc:357] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.408322 1188 raft_consensus.cc:383] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.408399 1188 raft_consensus.cc:738] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.408634 1188 consensus_queue.cc:260] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.409165 1188 ts_tablet_manager.cc:1428] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:20.409453 1188 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.412572 1439 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_94cf6a20190647dd87d3616b069b0243" } modify_external_catalogs: true
I20250110 02:51:20.412724 1439 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_94cf6a20190647dd87d3616b069b0243" } modify_external_catalogs: true
I20250110 02:51:20.416604 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9da4d8583ce74605a80da29ac030692a (DEFAULT_TABLE table=default.Table_34ce8eea83364d1ca7d8cfa100115b45 [id=4696b770f7514e2f9d5aab8e16ad9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.417050 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da4d8583ce74605a80da29ac030692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.421661 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_9cd2f808ffbc4122be3501fc1a25c768" } modify_external_catalogs: true
I20250110 02:51:20.422449 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_9cd2f808ffbc4122be3501fc1a25c768" } modify_external_catalogs: true
I20250110 02:51:20.424592 1449 master_service.cc:946] ReplaceTablet: received request to replace tablet f63d69d04b494a848653636c9e259105 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:20.424935 747 tablet_service.cc:1467] Processing CreateTablet for tablet 023b333c321a40dcb83a67ec5fb8ff0e (DEFAULT_TABLE table=default.Table_5e1c1dd369d94a688dabf41fda3795b6 [id=474dc245f50f45b4b43a1e6886a33c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.425410 741 tablet_service.cc:1467] Processing CreateTablet for tablet 89b5554630004cc6bb2574863f07cbd4 (DEFAULT_TABLE table=default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c [id=103f9b0512f343a29e3d8ba6de6071ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.426244 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaf7bbc981744f2bd78b97a3624448d (DEFAULT_TABLE table=default.Table_7fdb03687ec14267bfee2826ddb5a13b [id=14b73a8ae2744e1ea505735074b5c3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.426525 1188 tablet_bootstrap.cc:654] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.426878 892 tablet_service.cc:1467] Processing CreateTablet for tablet 023b333c321a40dcb83a67ec5fb8ff0e (DEFAULT_TABLE table=default.Table_5e1c1dd369d94a688dabf41fda3795b6 [id=474dc245f50f45b4b43a1e6886a33c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.427212 891 tablet_service.cc:1467] Processing CreateTablet for tablet 89b5554630004cc6bb2574863f07cbd4 (DEFAULT_TABLE table=default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c [id=103f9b0512f343a29e3d8ba6de6071ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.427790 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaf7bbc981744f2bd78b97a3624448d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.429931 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaf7bbc981744f2bd78b97a3624448d (DEFAULT_TABLE table=default.Table_7fdb03687ec14267bfee2826ddb5a13b [id=14b73a8ae2744e1ea505735074b5c3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.432648 1770 raft_consensus.cc:3054] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.433036 750 tablet_service.cc:1467] Processing CreateTablet for tablet d48a22b9642a442e83a3ecb7117d5c80 (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.433478 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b333c321a40dcb83a67ec5fb8ff0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.433962 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48a22b9642a442e83a3ecb7117d5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.437091 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b5554630004cc6bb2574863f07cbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.438787 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaf7bbc981744f2bd78b97a3624448d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.438935 1989 leader_election.cc:290] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.439111 996 leader_election.cc:304] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
W20250110 02:51:20.439667 996 leader_election.cc:343] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:20.439970 1989 raft_consensus.cc:2798] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.440222 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4f88a7002f413fbd63491fe8e1fe6d (DEFAULT_TABLE table=default.Table_2009672da4c945ec8fbbac6f5fe0988a [id=c409e93003184d2282188bd417dd3580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.440534 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4f88a7002f413fbd63491fe8e1fe6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.446800 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 9da4d8583ce74605a80da29ac030692a (DEFAULT_TABLE table=default.Table_34ce8eea83364d1ca7d8cfa100115b45 [id=4696b770f7514e2f9d5aab8e16ad9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.447125 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da4d8583ce74605a80da29ac030692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.450268 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 023b333c321a40dcb83a67ec5fb8ff0e (DEFAULT_TABLE table=default.Table_5e1c1dd369d94a688dabf41fda3795b6 [id=474dc245f50f45b4b43a1e6886a33c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.456732 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b333c321a40dcb83a67ec5fb8ff0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.458007 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 1eaf7bbc981744f2bd78b97a3624448d (DEFAULT_TABLE table=default.Table_7fdb03687ec14267bfee2826ddb5a13b [id=14b73a8ae2744e1ea505735074b5c3ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.458352 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eaf7bbc981744f2bd78b97a3624448d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.459295 1038 tablet_service.cc:1467] Processing CreateTablet for tablet f3e808cfa6df40478892465f7bf5a8e5 (DEFAULT_TABLE table=default.Table_a171d415e02a4113afd448d388d8f124 [id=fa45e9ce8e684b3191fb25c7b33246ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.459652 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 89b5554630004cc6bb2574863f07cbd4 (DEFAULT_TABLE table=default.Table_d7b0a52df5b84cbeb5d4f56f66e94a6c [id=103f9b0512f343a29e3d8ba6de6071ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.459223 1031 tablet_service.cc:1467] Processing CreateTablet for tablet b4f1782c46a44ed091f4c533226c71f2 (DEFAULT_TABLE table=default.Table_18eb7bac666944ddaf84da77edd4462b [id=5d3953c49f6f46b29e8ea2cd1f72f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.460839 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e808cfa6df40478892465f7bf5a8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.462142 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f1782c46a44ed091f4c533226c71f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.465106 894 tablet_service.cc:1467] Processing CreateTablet for tablet d48a22b9642a442e83a3ecb7117d5c80 (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.469488 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b5554630004cc6bb2574863f07cbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.469776 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48a22b9642a442e83a3ecb7117d5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.479668 1832 raft_consensus.cc:491] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.479832 1832 raft_consensus.cc:513] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.480418 1832 leader_election.cc:290] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.482017 1444 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e4c35b6a442b4b95a1341303de60226f" } modify_external_catalogs: true
I20250110 02:51:20.482252 1444 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e4c35b6a442b4b95a1341303de60226f" } modify_external_catalogs: true
I20250110 02:51:20.483235 1450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_108282f5803c488098a01d20b1e4f31a" } modify_external_catalogs: true
I20250110 02:51:20.483453 1450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_108282f5803c488098a01d20b1e4f31a" } modify_external_catalogs: true
I20250110 02:51:20.485172 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e183122d13a4c38988bf2fe6f7a3694 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:20.485458 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b333c321a40dcb83a67ec5fb8ff0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.485675 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7f1cbd8bd80f49cb9113143ecccdb8fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.486126 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f1cbd8bd80f49cb9113143ecccdb8fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.486362 1188 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.486544 1188 ts_tablet_manager.cc:1397] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.008s
I20250110 02:51:20.488104 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:20.451717 1184 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.481074 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.489487 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:20.505443 1449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet f63d69d04b494a848653636c9e259105
I20250110 02:51:20.505941 1449 catalog_manager.cc:6596] ReplaceTablet: tablet f63d69d04b494a848653636c9e259105 deleted and replaced by tablet 24034fb3da6e47978cdc9b02e714e30d
I20250110 02:51:20.506316 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_4c42814ef08441adadf295f4eb6b1fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.506834 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c42814ef08441adadf295f4eb6b1fc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.490592 1991 raft_consensus.cc:491] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.510094 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89b5554630004cc6bb2574863f07cbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.511744 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f15078fc22f4b32a48c7190b36e2192" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.511844 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.512046 1027 tablet_service.cc:1467] Processing CreateTablet for tablet d48a22b9642a442e83a3ecb7117d5c80 (DEFAULT_TABLE table=default.Table_f2fa05ad4d3d4931b4b42b8682420ce9 [id=91aba946a80142e9add85cb1739fc539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.512385 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48a22b9642a442e83a3ecb7117d5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.513028 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:20.513459 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet fc1324eeac9446279304f442e2fad488 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:20.511019 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.513612 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f63d69d04b494a848653636c9e259105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63d69d04b494a848653636c9e259105 already in progress: creating tablet
I20250110 02:51:20.511762 1181 tablet_bootstrap.cc:492] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
W20250110 02:51:20.515666 2003 log.cc:926] Time spent T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250110 02:51:20.487376 1188 raft_consensus.cc:357] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.518535 1181 ts_tablet_manager.cc:1397] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.016s
I20250110 02:51:20.522840 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc89802926543738cb04c75821ab693" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.530447 1061 raft_consensus.cc:1270] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:20.532330 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_b0925c3e58414c7a9582b4ede6d7b5b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:20.532859 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0925c3e58414c7a9582b4ede6d7b5b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:20.536710 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet fc1324eeac9446279304f442e2fad488
I20250110 02:51:20.537456 1456 catalog_manager.cc:6596] ReplaceTablet: tablet fc1324eeac9446279304f442e2fad488 deleted and replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1
I20250110 02:51:20.537914 1456 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_c8df69267ed3452192228c2106e18a47" } new_table_name: "default.Table_f3aa7e42b29e455aa70415be887fded5" modify_external_catalogs: true
I20250110 02:51:20.541644 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:20.543377 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:20.544698 1181 raft_consensus.cc:357] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.545009 1181 raft_consensus.cc:383] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.545812 1181 raft_consensus.cc:738] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.546501 1181 consensus_queue.cc:260] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.548102 1991 raft_consensus.cc:513] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:20.548569 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f63d69d04b494a848653636c9e259105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63d69d04b494a848653636c9e259105 already in progress: creating tablet
I20250110 02:51:20.548945 1991 leader_election.cc:290] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:20.549712 711 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.549813 711 leader_election.cc:304] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:20.550153 711 leader_election.cc:343] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.550249 711 leader_election.cc:304] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.550351 1188 raft_consensus.cc:383] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.550525 1188 raft_consensus.cc:738] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.550812 1184 ts_tablet_manager.cc:1397] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.013s
I20250110 02:51:20.550781 1188 consensus_queue.cc:260] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.551189 775 raft_consensus.cc:1270] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Refusing update from remote peer 85f9597077934f1a9814d284ed06d13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 02:51:20.552102 1184 raft_consensus.cc:357] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.552165 752 tablet_service.cc:1467] Processing CreateTablet for tablet e4726347afd649afa942d74df8c013fe (DEFAULT_TABLE table=default.Table_633f75bba62141e9a0075b23b2ec4bc0 [id=9b6434aff03e40a086c035ef1a731d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.552295 1184 raft_consensus.cc:383] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.552378 1184 raft_consensus.cc:738] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.552554 1184 consensus_queue.cc:260] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.552168 1991 raft_consensus.cc:2743] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.553090 1184 ts_tablet_manager.cc:1428] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:20.552838 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:20.554280 747 tablet_service.cc:1467] Processing CreateTablet for tablet 09c029deb4c54ce68d560059020f0bba (DEFAULT_TABLE table=default.Table_e55025c253874d759eae8dd088e59196 [id=db8c2210559d44c6bd41a41f4d49ee21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.554720 750 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:20.555126 1184 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.556697 1184 tablet_bootstrap.cc:654] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.557674 1957 consensus_queue.cc:1035] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
W20250110 02:51:20.558867 711 leader_election.cc:343] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.558975 711 leader_election.cc:304] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.559263 1991 raft_consensus.cc:2743] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.559405 751 tablet_service.cc:1467] Processing CreateTablet for tablet dc1901385a534e5dad20954b763329a0 (DEFAULT_TABLE table=default.Table_0de0a6680c0f4531a75432a618576660 [id=72c128c9bf8849638e64f4eb99ec4c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.559546 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4726347afd649afa942d74df8c013fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.559746 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1901385a534e5dad20954b763329a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.559463 1991 raft_consensus.cc:2743] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.562839 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c029deb4c54ce68d560059020f0bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.562789 894 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:20.567051 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f63d69d04b494a848653636c9e259105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63d69d04b494a848653636c9e259105 already in progress: creating tablet
W20250110 02:51:20.567315 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:20.568850 1991 raft_consensus.cc:491] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.568958 1991 raft_consensus.cc:513] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.569475 1991 leader_election.cc:290] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.573464 895 tablet_service.cc:1467] Processing CreateTablet for tablet dc1901385a534e5dad20954b763329a0 (DEFAULT_TABLE table=default.Table_0de0a6680c0f4531a75432a618576660 [id=72c128c9bf8849638e64f4eb99ec4c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.573837 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1901385a534e5dad20954b763329a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.575572 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.576323 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
W20250110 02:51:20.576444 709 leader_election.cc:343] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.577044 1188 ts_tablet_manager.cc:1428] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.090s user 0.001s sys 0.001s
I20250110 02:51:20.577384 1188 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.579162 1188 tablet_bootstrap.cc:654] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.579500 853 leader_election.cc:304] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.579751 889 tablet_service.cc:1467] Processing CreateTablet for tablet e4726347afd649afa942d74df8c013fe (DEFAULT_TABLE table=default.Table_633f75bba62141e9a0075b23b2ec4bc0 [id=9b6434aff03e40a086c035ef1a731d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.581032 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 09c029deb4c54ce68d560059020f0bba (DEFAULT_TABLE table=default.Table_e55025c253874d759eae8dd088e59196 [id=db8c2210559d44c6bd41a41f4d49ee21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.581491 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c029deb4c54ce68d560059020f0bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.591524 1181 ts_tablet_manager.cc:1428] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250110 02:51:20.594687 1028 tablet_service.cc:1467] Processing CreateTablet for tablet dc1901385a534e5dad20954b763329a0 (DEFAULT_TABLE table=default.Table_0de0a6680c0f4531a75432a618576660 [id=72c128c9bf8849638e64f4eb99ec4c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.595067 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1901385a534e5dad20954b763329a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:20.596719 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:20.598585 1991 raft_consensus.cc:491] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.598745 1991 raft_consensus.cc:513] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.599344 1991 leader_election.cc:290] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.599949 1181 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.594101 1770 raft_consensus.cc:513] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.611017 1037 tablet_service.cc:1467] Processing CreateTablet for tablet e4726347afd649afa942d74df8c013fe (DEFAULT_TABLE table=default.Table_633f75bba62141e9a0075b23b2ec4bc0 [id=9b6434aff03e40a086c035ef1a731d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.611922 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.612210 996 consensus_peers.cc:487] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:20.612315 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 49613.7us
I20250110 02:51:20.614364 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:20.614586 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]). Expected version 1 got 0
I20250110 02:51:20.614753 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:20.615103 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:20.615248 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:20.615361 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:20.615483 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:20.615665 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
W20250110 02:51:20.615803 711 leader_election.cc:343] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.615918 711 leader_election.cc:304] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:20.615978 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:20.616235 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:20.616411 1991 raft_consensus.cc:2743] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.616415 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:20.616596 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09]). Expected version 3 got 0
I20250110 02:51:20.616852 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:20.617270 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:20.617482 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:20.617955 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:20.618155 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
W20250110 02:51:20.620891 2026 log.cc:926] Time spent T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250110 02:51:20.620947 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.621392 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4726347afd649afa942d74df8c013fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.622186 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.623122 1770 leader_election.cc:290] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:20.623347 996 leader_election.cc:343] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:20.624356 1440 catalog_manager.cc:5526] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.624920 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:20.625157 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:20.625598 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]). Expected version 1 got 0
I20250110 02:51:20.625929 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:20.626225 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:20.626600 1440 catalog_manager.cc:5526] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
W20250110 02:51:20.629321 711 leader_election.cc:343] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:20.629513 711 leader_election.cc:343] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.629555 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:20.629947 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:20.630887 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:20.631096 1440 catalog_manager.cc:5526] T 8bc89802926543738cb04c75821ab693 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:20.631723 1440 catalog_manager.cc:5526] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.632133 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:20.632368 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_c8df69267ed3452192228c2106e18a47 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 1 got 0
I20250110 02:51:20.632620 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:20.633100 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:20.633329 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:20.633647 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:20.640003 1440 catalog_manager.cc:5526] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:20.640574 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 1 got 0
I20250110 02:51:20.640766 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:20.640964 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:20.641301 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:20.641510 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:20.641708 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:20.642020 1440 catalog_manager.cc:5526] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.642943 2033 tablet.cc:1721] T 3352f8378b9846a3854086293f2d4e15 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:20.642367 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:20.645937 1953 raft_consensus.cc:491] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.646152 1953 raft_consensus.cc:513] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.645486 1440 catalog_manager.cc:5526] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.646745 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:20.646919 1953 leader_election.cc:290] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.647121 1440 catalog_manager.cc:5526] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.647490 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
W20250110 02:51:20.648026 853 leader_election.cc:343] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:20.648253 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:20.648581 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:20.648615 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ae68e4f0254ef38aafd5a1f8f88a54" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.648782 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
I20250110 02:51:20.648829 1059 raft_consensus.cc:2463] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.649315 853 leader_election.cc:304] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.649741 1953 raft_consensus.cc:2798] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.649829 1953 raft_consensus.cc:491] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.649914 1953 raft_consensus.cc:3054] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.652217 1181 tablet_bootstrap.cc:654] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.652837 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:20.653046 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:20.653214 1953 raft_consensus.cc:513] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.653264 1440 catalog_manager.cc:5526] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:20.653625 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:20.653836 1440 catalog_manager.cc:5526] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:20.654175 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:20.655212 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ae68e4f0254ef38aafd5a1f8f88a54" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:20.655498 1059 raft_consensus.cc:3054] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.656482 1440 catalog_manager.cc:5526] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } }
I20250110 02:51:20.656800 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:20.656970 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:20.659126 1059 raft_consensus.cc:2463] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.659628 853 leader_election.cc:304] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:20.659883 1916 raft_consensus.cc:2798] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:20.660912 853 leader_election.cc:343] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:20.661136 1953 leader_election.cc:290] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.704449 1989 raft_consensus.cc:695] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:20.704834 1989 consensus_queue.cc:237] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.714267 894 tablet_service.cc:1467] Processing CreateTablet for tablet 09c029deb4c54ce68d560059020f0bba (DEFAULT_TABLE table=default.Table_e55025c253874d759eae8dd088e59196 [id=db8c2210559d44c6bd41a41f4d49ee21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.718556 1726 raft_consensus.cc:2743] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.719379 1832 consensus_queue.cc:1035] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.139s
I20250110 02:51:20.726164 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.726518 1957 consensus_queue.cc:1035] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250110 02:51:20.729342 2031 tablet.cc:1721] T 3352f8378b9846a3854086293f2d4e15 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:20.729619 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4726347afd649afa942d74df8c013fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.730424 1184 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.730551 1184 ts_tablet_manager.cc:1397] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.175s user 0.006s sys 0.003s
I20250110 02:51:20.730767 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc77ab5b074cc3af95c325cd5caa8e (DEFAULT_TABLE table=default.Table_4c42814ef08441adadf295f4eb6b1fc8 [id=37bfab85a0964d9e9c011e6553852149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.731210 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc77ab5b074cc3af95c325cd5caa8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.731305 1184 raft_consensus.cc:357] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.731505 1184 raft_consensus.cc:383] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.731568 1184 raft_consensus.cc:738] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.732793 1181 tablet_bootstrap.cc:492] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:20.732973 1181 ts_tablet_manager.cc:1397] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.000s
I20250110 02:51:20.735327 2030 tablet.cc:1721] T 3352f8378b9846a3854086293f2d4e15 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:20.739711 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc77ab5b074cc3af95c325cd5caa8e (DEFAULT_TABLE table=default.Table_4c42814ef08441adadf295f4eb6b1fc8 [id=37bfab85a0964d9e9c011e6553852149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.740141 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc77ab5b074cc3af95c325cd5caa8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.741446 1181 raft_consensus.cc:357] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.741643 1181 raft_consensus.cc:383] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.741722 1181 raft_consensus.cc:738] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.741976 1184 consensus_queue.cc:260] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.744007 1181 consensus_queue.cc:260] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.744542 1181 ts_tablet_manager.cc:1428] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250110 02:51:20.746735 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af3d61366b2432db53b18f2b7bb49d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:20.746989 853 consensus_peers.cc:487] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da8f247c862e4cb68452e230857ee6b0. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.752111 1184 ts_tablet_manager.cc:1428] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20250110 02:51:20.752377 1184 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.752710 1991 raft_consensus.cc:491] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.752821 1991 raft_consensus.cc:513] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.752977 1680 consensus_queue.cc:1035] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.753029 2025 consensus_queue.cc:1035] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.753329 1991 leader_election.cc:290] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.753485 2025 consensus_queue.cc:1035] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.753952 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.754448 711 leader_election.cc:343] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.757198 1953 raft_consensus.cc:491] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.757449 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b88daa882e8b4b60a4d5eaf2759747ef (DEFAULT_TABLE table=default.Table_7f1cbd8bd80f49cb9113143ecccdb8fa [id=a49b9c0ebe4d4b5a93f38a05656e75aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.757828 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88daa882e8b4b60a4d5eaf2759747ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.758636 743 tablet_service.cc:1467] Processing CreateTablet for tablet b88daa882e8b4b60a4d5eaf2759747ef (DEFAULT_TABLE table=default.Table_7f1cbd8bd80f49cb9113143ecccdb8fa [id=a49b9c0ebe4d4b5a93f38a05656e75aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.759260 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88daa882e8b4b60a4d5eaf2759747ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.760629 1726 raft_consensus.cc:491] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.762882 1726 raft_consensus.cc:513] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.763322 917 raft_consensus.cc:3054] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.765425 1022 tablet_service.cc:1467] Processing CreateTablet for tablet e2723e18948d4248b24135443822a956 (DEFAULT_TABLE table=default.Table_de1fbb2884e2436193565eff3bc88357 [id=d09924b343ce4da8bf8f45f51fbfff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.766218 755 tablet_service.cc:1467] Processing CreateTablet for tablet e2723e18948d4248b24135443822a956 (DEFAULT_TABLE table=default.Table_de1fbb2884e2436193565eff3bc88357 [id=d09924b343ce4da8bf8f45f51fbfff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.766260 1953 raft_consensus.cc:513] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.766558 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2723e18948d4248b24135443822a956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.766817 1953 leader_election.cc:290] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.767067 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2723e18948d4248b24135443822a956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.769691 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c029deb4c54ce68d560059020f0bba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:20.772889 853 consensus_peers.cc:487] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.773591 2040 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250110 02:51:20.774441 996 consensus_peers.cc:487] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:20.780870 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.781050 776 raft_consensus.cc:2463] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.780876 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:20.781342 775 raft_consensus.cc:2388] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:20.783288 1991 raft_consensus.cc:491] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.783403 1991 raft_consensus.cc:513] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.783986 1991 leader_election.cc:290] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.791929 1184 tablet_bootstrap.cc:654] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.786482 1060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.795583 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3510e86314144a2ca012aacf3d78eacc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.795583 1060 raft_consensus.cc:2463] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:20.796114 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4778347f054b86af697dbe3b287263" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.796682 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23 (DEFAULT_TABLE table=default.Table_0943d6c040ee49e1b0ccdc1ca19457af [id=09d2d2433c1f47588425421505339e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.797195 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.801071 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 4cba2139e332432b982d02db24f5e87e (DEFAULT_TABLE table=default.Table_9b414f28d5514d17a68f7f198a2054d7 [id=9141120691cc4a92b50315fb70aa66aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.801456 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cba2139e332432b982d02db24f5e87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.803007 1041 tablet_service.cc:1467] Processing CreateTablet for tablet a196c8b5a07640d7a4ae33024c322f5f (DEFAULT_TABLE table=default.Table_bdaba3ed414148628957dbf5e4462911 [id=f533a551f413448a95faab72f4e0e1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.804668 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a196c8b5a07640d7a4ae33024c322f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.806743 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3aee71c51a96482fbb527576867bbaa9 (DEFAULT_TABLE table=default.Table_aad85d64a3ff4f87b6c4f767f39de7df [id=c9a367ed70c742a18d7dc76e72b37300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.807019 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aee71c51a96482fbb527576867bbaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.807559 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 89e3bcb739b740b88e98f2717f58e0c2 (DEFAULT_TABLE table=default.Table_829a8c82f1a44c3b9b1b67047764be79 [id=b800401c0e744bd48fda3ee1c851862e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.808166 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e3bcb739b740b88e98f2717f58e0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.809010 752 tablet_service.cc:1467] Processing CreateTablet for tablet 89e3bcb739b740b88e98f2717f58e0c2 (DEFAULT_TABLE table=default.Table_829a8c82f1a44c3b9b1b67047764be79 [id=b800401c0e744bd48fda3ee1c851862e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.809428 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e3bcb739b740b88e98f2717f58e0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.793465 755 tablet_service.cc:1467] Processing CreateTablet for tablet a196c8b5a07640d7a4ae33024c322f5f (DEFAULT_TABLE table=default.Table_bdaba3ed414148628957dbf5e4462911 [id=f533a551f413448a95faab72f4e0e1fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:20.796180 711 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.813515 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a196c8b5a07640d7a4ae33024c322f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.798169 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23 (DEFAULT_TABLE table=default.Table_0943d6c040ee49e1b0ccdc1ca19457af [id=09d2d2433c1f47588425421505339e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.801280 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4cba2139e332432b982d02db24f5e87e (DEFAULT_TABLE table=default.Table_9b414f28d5514d17a68f7f198a2054d7 [id=9141120691cc4a92b50315fb70aa66aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.815762 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.817976 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cba2139e332432b982d02db24f5e87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.827270 1184 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:20.827452 1184 ts_tablet_manager.cc:1397] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.003s
I20250110 02:51:20.828338 1184 raft_consensus.cc:357] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.828594 1184 raft_consensus.cc:383] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.828631 1184 raft_consensus.cc:738] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.828833 1184 consensus_queue.cc:260] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.829651 1184 ts_tablet_manager.cc:1428] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:20.830135 1184 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:20.831948 1184 tablet_bootstrap.cc:654] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.805667 743 tablet_service.cc:1467] Processing CreateTablet for tablet 3aee71c51a96482fbb527576867bbaa9 (DEFAULT_TABLE table=default.Table_aad85d64a3ff4f87b6c4f767f39de7df [id=c9a367ed70c742a18d7dc76e72b37300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.833006 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aee71c51a96482fbb527576867bbaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.840939 853 leader_election.cc:304] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:20.842031 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb (DEFAULT_TABLE table=default.Table_bd274cbe850641b69ba5a4b6fa3f2474 [id=378a8ab24eee458b89ddf29fae4d39a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.842468 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.841521 1726 leader_election.cc:290] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.852365 752 tablet_service.cc:1467] Processing CreateTablet for tablet 24034fb3da6e47978cdc9b02e714e30d (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:20.856613 996 consensus_peers.cc:487] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:20.860968 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 49411us
I20250110 02:51:20.861729 853 leader_election.cc:304] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.862028 1953 raft_consensus.cc:2798] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.862125 1953 raft_consensus.cc:491] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.862224 1953 raft_consensus.cc:3054] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.863092 1726 raft_consensus.cc:3054] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.865542 917 raft_consensus.cc:2463] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:20.865768 1726 raft_consensus.cc:2743] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:20.870198 1953 raft_consensus.cc:513] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.871481 1953 leader_election.cc:290] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.873540 747 tablet_service.cc:1467] Processing CreateTablet for tablet 291852e862c24f589d16599a812d0992 (DEFAULT_TABLE table=default.Table_af9287df67924374975e9318755998f4 [id=c60d14c6bea94079a3401319303ccba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.877027 1181 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:20.878103 1916 raft_consensus.cc:695] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:20.866220 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb (DEFAULT_TABLE table=default.Table_bd274cbe850641b69ba5a4b6fa3f2474 [id=378a8ab24eee458b89ddf29fae4d39a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.882143 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 24034fb3da6e47978cdc9b02e714e30d (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.882458 911 raft_consensus.cc:3054] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.886269 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24034fb3da6e47978cdc9b02e714e30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.887122 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291852e862c24f589d16599a812d0992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.889657 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.891518 1916 consensus_queue.cc:237] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.892711 2030 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:20.892413 994 leader_election.cc:304] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.894562 1849 raft_consensus.cc:2798] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:20.894713 1849 raft_consensus.cc:695] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:20.895030 1849 consensus_queue.cc:237] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.896520 2025 consensus_queue.cc:1035] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:20.899897 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24034fb3da6e47978cdc9b02e714e30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.923152 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 291852e862c24f589d16599a812d0992 (DEFAULT_TABLE table=default.Table_af9287df67924374975e9318755998f4 [id=c60d14c6bea94079a3401319303ccba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:20.923662 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291852e862c24f589d16599a812d0992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:20.924135 1991 raft_consensus.cc:491] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.924264 1991 raft_consensus.cc:513] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.927212 1991 leader_election.cc:290] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.929136 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.933041 711 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.934357 1991 raft_consensus.cc:491] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.934645 1991 raft_consensus.cc:513] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.935230 1991 leader_election.cc:290] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.935998 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:20.936968 711 leader_election.cc:343] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.939262 1181 tablet_bootstrap.cc:654] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.951130 1957 raft_consensus.cc:491] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.951315 1957 raft_consensus.cc:513] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.951965 1957 leader_election.cc:290] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.952945 1989 consensus_queue.cc:1035] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250110 02:51:20.973639 1188 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:20.973801 1188 ts_tablet_manager.cc:1397] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.397s user 0.011s sys 0.020s
I20250110 02:51:20.975790 1991 raft_consensus.cc:491] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.975944 1991 raft_consensus.cc:513] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.976732 1991 leader_election.cc:290] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.977659 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7397e41f7374e9fb1621dbcf30af703" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.977777 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.978303 711 leader_election.cc:343] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.983186 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.983186 914 raft_consensus.cc:3054] T 8bc89802926543738cb04c75821ab693 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.986804 1991 raft_consensus.cc:491] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:20.986953 1991 raft_consensus.cc:513] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.987489 1991 leader_election.cc:290] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:20.989010 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:20.989387 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3510e86314144a2ca012aacf3d78eacc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.989557 711 leader_election.cc:343] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:20.990612 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.991479 1990 raft_consensus.cc:491] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:20.991704 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 129955us
I20250110 02:51:20.991940 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.993144 1188 raft_consensus.cc:357] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.993340 1188 raft_consensus.cc:383] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:20.993434 1188 raft_consensus.cc:738] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:20.993639 1188 consensus_queue.cc:260] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:20.994168 1188 ts_tablet_manager.cc:1428] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250110 02:51:20.994252 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:20.994406 775 raft_consensus.cc:3054] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:20.994467 1188 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:20.995921 1188 tablet_bootstrap.cc:654] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:20.991672 1990 raft_consensus.cc:513] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:20.996824 775 raft_consensus.cc:2463] T 397f3261b76a4c82a3c265e68662841f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:20.997152 1990 leader_election.cc:290] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:20.997273 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7397e41f7374e9fb1621dbcf30af703" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:20.997905 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3ce95413df40369ee599176f91b892" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:20.997941 996 leader_election.cc:343] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8c3ce95413df40369ee599176f91b892
I20250110 02:51:20.998072 913 raft_consensus.cc:2463] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:20.998668 994 leader_election.cc:304] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:20.999454 1770 raft_consensus.cc:2798] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:20.999624 1770 raft_consensus.cc:491] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:20.999761 1770 raft_consensus.cc:3054] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:21.002099 709 leader_election.cc:343] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.002197 709 leader_election.cc:304] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.002465 1991 raft_consensus.cc:2743] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.002020 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.002764 775 raft_consensus.cc:2463] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:21.003216 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397f3261b76a4c82a3c265e68662841f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:21.003432 1059 raft_consensus.cc:3054] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.003769 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.004926 1991 raft_consensus.cc:491] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.005044 1991 raft_consensus.cc:513] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.005348 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2 (DEFAULT_TABLE table=default.Table_7ac37bfbe9d7409cbd07212eceafa582 [id=81d5e73bc81c4903817b104235cb8eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.005548 1991 leader_election.cc:290] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.006254 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.006690 711 leader_election.cc:343] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:21.010845 709 leader_election.cc:343] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.010937 709 leader_election.cc:304] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.011226 1991 raft_consensus.cc:2743] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.011238 1680 consensus_queue.cc:1035] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:21.011657 709 leader_election.cc:343] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.011768 709 leader_election.cc:304] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.012106 1991 raft_consensus.cc:2743] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:21.012493 709 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.012579 709 leader_election.cc:304] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.013053 1991 raft_consensus.cc:2743] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:21.013727 709 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.013844 709 leader_election.cc:304] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.014974 1991 raft_consensus.cc:2743] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.015385 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:21.015554 1061 raft_consensus.cc:2463] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:21.016633 1184 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.016768 1184 ts_tablet_manager.cc:1397] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.187s user 0.034s sys 0.032s
W20250110 02:51:21.016856 709 leader_election.cc:343] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.016945 709 leader_election.cc:304] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.017242 1991 raft_consensus.cc:2743] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.017573 1184 raft_consensus.cc:357] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.017618 853 leader_election.cc:304] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
W20250110 02:51:21.017655 709 leader_election.cc:343] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.017761 709 leader_election.cc:304] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.017985 1991 raft_consensus.cc:2743] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.018117 1184 raft_consensus.cc:383] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.018218 1184 raft_consensus.cc:738] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.018496 1957 raft_consensus.cc:491] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.018474 1184 consensus_queue.cc:260] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.018678 1957 raft_consensus.cc:513] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.018962 1184 ts_tablet_manager.cc:1428] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.019246 1957 leader_election.cc:290] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.019328 1184 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.019512 1957 consensus_queue.cc:1035] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.019927 1957 consensus_queue.cc:1035] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.021041 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2 (DEFAULT_TABLE table=default.Table_7ac37bfbe9d7409cbd07212eceafa582 [id=81d5e73bc81c4903817b104235cb8eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.021432 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.021665 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:21.021865 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
I20250110 02:51:21.022006 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:21.023701 1184 tablet_bootstrap.cc:654] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.024711 747 tablet_service.cc:1467] Processing CreateTablet for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e (DEFAULT_TABLE table=default.Table_4ef1477d7e314ddbaa6fd875538bc552 [id=d7c9b51aed334036850b6f60bbcb4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.024829 1181 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.024982 1181 ts_tablet_manager.cc:1397] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.159s user 0.006s sys 0.006s
I20250110 02:51:21.025031 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.025700 1181 raft_consensus.cc:357] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.025902 1181 raft_consensus.cc:383] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.025996 1181 raft_consensus.cc:738] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.026264 1181 consensus_queue.cc:260] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.026873 1181 ts_tablet_manager.cc:1428] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.032526 1022 tablet_service.cc:1467] Processing CreateTablet for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.032918 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd02e9f8cd0748a0a6a3202fbe390ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.033862 1770 raft_consensus.cc:513] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.035426 1770 leader_election.cc:290] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:21.035758 996 leader_election.cc:343] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8c3ce95413df40369ee599176f91b892
I20250110 02:51:21.039482 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3ce95413df40369ee599176f91b892" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.039754 912 raft_consensus.cc:3054] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.041716 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e (DEFAULT_TABLE table=default.Table_4ef1477d7e314ddbaa6fd875538bc552 [id=d7c9b51aed334036850b6f60bbcb4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.042073 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.042358 1181 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.043920 743 tablet_service.cc:1467] Processing CreateTablet for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.046720 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd02e9f8cd0748a0a6a3202fbe390ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.048216 2059 consensus_queue.cc:1035] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.048687 912 raft_consensus.cc:2463] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.048815 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:21.049172 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
W20250110 02:51:21.050789 996 consensus_peers.cc:487] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fb5412b78104d8085878c58b191ab27. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.055763 1059 raft_consensus.cc:2463] T 397f3261b76a4c82a3c265e68662841f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:21.060420 1181 tablet_bootstrap.cc:654] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.063735 1184 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.063879 1184 ts_tablet_manager.cc:1397] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.005s
I20250110 02:51:21.064955 1184 raft_consensus.cc:357] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.065150 1184 raft_consensus.cc:383] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.065200 1184 raft_consensus.cc:738] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.065402 1184 consensus_queue.cc:260] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.066659 1184 ts_tablet_manager.cc:1428] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:21.068065 994 leader_election.cc:304] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:21.068528 1770 raft_consensus.cc:2798] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.068136 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:21.068674 1770 raft_consensus.cc:695] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.068914 1770 consensus_queue.cc:237] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.069307 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:21.069991 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:21.072785 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:21.074218 1181 tablet_bootstrap.cc:492] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.074393 1181 ts_tablet_manager.cc:1397] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250110 02:51:21.074712 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:21.074944 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_055d44655c2743f3907923f2f49a33af [id=6985bca790e14c79927bac8621fb5d44]). Expected version 1 got 0
I20250110 02:51:21.075194 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:21.075659 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:21.077358 1181 raft_consensus.cc:357] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.077603 1181 raft_consensus.cc:383] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.077688 1181 raft_consensus.cc:738] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.077703 1184 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.077497 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:21.078125 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:21.078423 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:21.079782 1184 tablet_bootstrap.cc:654] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.080035 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:21.080708 1181 consensus_queue.cc:260] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.081153 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:21.081436 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa [id=996c474b298141e48b7a828a367d0e09]). Expected version 3 got 0
I20250110 02:51:21.082321 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:21.087291 894 tablet_service.cc:1467] Processing CreateTablet for tablet 2abc77ab5b074cc3af95c325cd5caa8e (DEFAULT_TABLE table=default.Table_4c42814ef08441adadf295f4eb6b1fc8 [id=37bfab85a0964d9e9c011e6553852149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.087754 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abc77ab5b074cc3af95c325cd5caa8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.087754 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:21.088199 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:21.088368 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:21.088662 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_c8df69267ed3452192228c2106e18a47 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 1 got 0
I20250110 02:51:21.088905 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:21.089053 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:21.089221 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:21.089423 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:21.089617 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:21.090200 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:21.090652 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:21.090919 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:21.091078 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:21.086055 1181 ts_tablet_manager.cc:1428] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20250110 02:51:21.091565 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:21.092614 1181 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.098047 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:21.098531 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:21.098583 1184 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.098733 1184 ts_tablet_manager.cc:1397] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250110 02:51:21.098745 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:21.098999 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:21.099208 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:21.099360 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:21.099493 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:21.099386 1184 raft_consensus.cc:357] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.099678 1184 raft_consensus.cc:383] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.099728 1184 raft_consensus.cc:738] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.099949 1184 consensus_queue.cc:260] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.100054 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:21.100262 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:21.100471 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:21.100916 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_618cae6951b649328aab854b7efa2a14 [id=38e6ddea3b604a84b097de85815029ba]). Expected version 1 got 0
I20250110 02:51:21.101121 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:21.101166 1184 ts_tablet_manager.cc:1428] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:21.101270 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 1 got 0
I20250110 02:51:21.101609 889 tablet_service.cc:1467] Processing CreateTablet for tablet b88daa882e8b4b60a4d5eaf2759747ef (DEFAULT_TABLE table=default.Table_7f1cbd8bd80f49cb9113143ecccdb8fa [id=a49b9c0ebe4d4b5a93f38a05656e75aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.101956 1184 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.101967 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88daa882e8b4b60a4d5eaf2759747ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.103448 1184 tablet_bootstrap.cc:654] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.103832 1181 tablet_bootstrap.cc:654] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.107862 2050 raft_consensus.cc:2798] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.107937 2050 raft_consensus.cc:491] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.107990 2050 raft_consensus.cc:3054] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.101539 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:21.110299 2050 raft_consensus.cc:513] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.110603 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:21.111033 2050 leader_election.cc:290] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.111137 1444 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet bc8444c1381a45d6b2fee30da971bd47
W20250110 02:51:21.111280 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 93699.6us
I20250110 02:51:21.111593 2050 consensus_queue.cc:1035] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.111678 1444 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 32) took 1225 ms. Trace:
I20250110 02:51:21.111785 1444 rpcz_store.cc:276] 0110 02:51:19.886120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.481827 (+595707us) service_pool.cc:225] Handling call
0110 02:51:20.482366 (+ 539us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.482525 (+ 159us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:20.482582 (+ 57us) catalog_manager.cc:2762] Locking tablets
0110 02:51:20.482727 (+ 145us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:20.512017 (+ 29290us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:20.512087 (+ 70us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:21.111056 (+598969us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:21.111642 (+ 586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.484026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.484132 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.484148 (+ 16us) write_op.cc:435] Acquired schema lock
0110 02:51:20.484150 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.484244 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.484253 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.484257 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:20.484335 (+ 78us) tablet.cc:698] Row locks acquired
0110 02:51:20.484337 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.484367 (+ 30us) write_op.cc:270] Start()
0110 02:51:20.484389 (+ 22us) write_op.cc:276] Timestamp: P: 1736477480484292 usec, L: 0
0110 02:51:20.484392 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.484548 (+ 156us) log.cc:844] Serialized 725 byte log entry
0110 02:51:20.511482 (+ 26934us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.511690 (+ 208us) write_op.cc:301] APPLY: starting
0110 02:51:20.511701 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.511730 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.511758 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.511806 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.511828 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:20.511834 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.511881 (+ 47us) log.cc:844] Serialized 29 byte log entry
0110 02:51:20.511925 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.511936 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:20.511941 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.512001 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.512007 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":304,"apply.run_wall_time_us":303,"num_ops":2,"prepare.queue_time_us":640,"prepare.run_cpu_time_us":597,"prepare.run_wall_time_us":597,"raft.queue_time_us":636,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":27066}]]}
I20250110 02:51:21.112797 889 tablet_service.cc:1467] Processing CreateTablet for tablet e2723e18948d4248b24135443822a956 (DEFAULT_TABLE table=default.Table_de1fbb2884e2436193565eff3bc88357 [id=d09924b343ce4da8bf8f45f51fbfff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.113197 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2723e18948d4248b24135443822a956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.115582 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:21.115959 743 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:21.117184 1181 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.117373 1181 ts_tablet_manager.cc:1397] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250110 02:51:21.118578 1181 raft_consensus.cc:357] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.118817 1181 raft_consensus.cc:383] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.118880 1181 raft_consensus.cc:738] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.119155 1181 consensus_queue.cc:260] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.120406 853 leader_election.cc:304] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:21.120769 2050 raft_consensus.cc:2798] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.120925 2050 raft_consensus.cc:695] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:21.121219 2050 consensus_queue.cc:237] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.123694 895 tablet_service.cc:1467] Processing CreateTablet for tablet a196c8b5a07640d7a4ae33024c322f5f (DEFAULT_TABLE table=default.Table_bdaba3ed414148628957dbf5e4462911 [id=f533a551f413448a95faab72f4e0e1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.124047 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a196c8b5a07640d7a4ae33024c322f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.124305 1181 ts_tablet_manager.cc:1428] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250110 02:51:21.126739 1181 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.128638 1181 tablet_bootstrap.cc:654] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.130903 895 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23 (DEFAULT_TABLE table=default.Table_0943d6c040ee49e1b0ccdc1ca19457af [id=09d2d2433c1f47588425421505339e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.131362 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc2f8b9c8344a8080f92fd7ceb51e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.135581 2033 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:21.137918 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4cba2139e332432b982d02db24f5e87e (DEFAULT_TABLE table=default.Table_9b414f28d5514d17a68f7f198a2054d7 [id=9141120691cc4a92b50315fb70aa66aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.138224 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cba2139e332432b982d02db24f5e87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.139550 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 8d1f2acd39be40e4b71cc0dcf68e7567
I20250110 02:51:21.140017 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 30) took 1254 ms. Trace:
I20250110 02:51:21.140136 1448 rpcz_store.cc:276] 0110 02:51:19.885970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.421487 (+535517us) service_pool.cc:225] Handling call
0110 02:51:20.423553 (+ 2066us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.423687 (+ 134us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:20.423742 (+ 55us) catalog_manager.cc:2762] Locking tablets
0110 02:51:20.424032 (+ 290us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:20.504824 (+ 80792us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:20.504892 (+ 68us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:21.139458 (+634566us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:21.139987 (+ 529us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.434486 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.434553 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.434557 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:20.434559 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.434634 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.434642 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.434646 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:20.434724 (+ 78us) tablet.cc:698] Row locks acquired
0110 02:51:20.434726 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.434749 (+ 23us) write_op.cc:270] Start()
0110 02:51:20.434770 (+ 21us) write_op.cc:276] Timestamp: P: 1736477480434681 usec, L: 0
0110 02:51:20.434773 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.434886 (+ 113us) log.cc:844] Serialized 725 byte log entry
0110 02:51:20.503826 (+ 68940us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.504500 (+ 674us) write_op.cc:301] APPLY: starting
0110 02:51:20.504510 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.504538 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.504558 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.504603 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.504629 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:20.504635 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.504687 (+ 52us) log.cc:844] Serialized 29 byte log entry
0110 02:51:20.504725 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.504737 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:20.504742 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.504809 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.504813 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":623,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":300,"num_ops":2,"prepare.queue_time_us":9907,"prepare.run_cpu_time_us":464,"prepare.run_wall_time_us":464,"raft.queue_time_us":805,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":2,"replication_time_us":69035}]]}
I20250110 02:51:21.140688 1184 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.140847 1184 ts_tablet_manager.cc:1397] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.001s
I20250110 02:51:21.140974 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_b031e9abfbe345358123f63c277edb99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.141520 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b031e9abfbe345358123f63c277edb99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.141628 1184 raft_consensus.cc:357] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.141808 1184 raft_consensus.cc:383] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.141845 1184 raft_consensus.cc:738] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.142503 1184 consensus_queue.cc:260] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.142577 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:21.143014 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:21.143114 1184 ts_tablet_manager.cc:1428] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250110 02:51:21.143193 1441 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac5dab241cd74f329a57e6f19e95a7b6 consensus state has no leader
I20250110 02:51:21.143481 1184 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:21.143569 1457 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e consensus state has no leader
I20250110 02:51:21.145335 1184 tablet_bootstrap.cc:654] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.148633 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:21.149103 1450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 33) took 1262 ms. Trace:
I20250110 02:51:21.149188 1450 rpcz_store.cc:276] 0110 02:51:19.886231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.483031 (+596800us) service_pool.cc:225] Handling call
0110 02:51:20.483584 (+ 553us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.483733 (+ 149us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:20.483792 (+ 59us) catalog_manager.cc:2762] Locking tablets
0110 02:51:20.483945 (+ 153us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:20.514597 (+ 30652us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:20.514670 (+ 73us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:21.148597 (+633927us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:21.149032 (+ 435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.484645 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.484720 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.484724 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:20.484726 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.484815 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.484823 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.484827 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:20.484899 (+ 72us) tablet.cc:698] Row locks acquired
0110 02:51:20.484902 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:20.484925 (+ 23us) write_op.cc:270] Start()
0110 02:51:20.484947 (+ 22us) write_op.cc:276] Timestamp: P: 1736477480484852 usec, L: 0
0110 02:51:20.484950 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.485074 (+ 124us) log.cc:844] Serialized 725 byte log entry
0110 02:51:20.511537 (+ 26463us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.512052 (+ 515us) write_op.cc:301] APPLY: starting
0110 02:51:20.512060 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.512091 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.512107 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.512145 (+ 38us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.512164 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:20.512170 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.512210 (+ 40us) log.cc:844] Serialized 29 byte log entry
0110 02:51:20.512262 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.512273 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:20.512278 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.512333 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.512336 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":491,"apply.run_cpu_time_us":271,"apply.run_wall_time_us":270,"num_ops":2,"prepare.queue_time_us":75,"prepare.run_cpu_time_us":488,"prepare.run_wall_time_us":487,"raft.queue_time_us":151,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":26572}]]}
I20250110 02:51:21.149890 1450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e8d197a297e647b5ad3b516ff8054704" } new_table_name: "default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed" modify_external_catalogs: true
I20250110 02:51:21.151309 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:21.151733 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:21.152141 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3aee71c51a96482fbb527576867bbaa9 (DEFAULT_TABLE table=default.Table_aad85d64a3ff4f87b6c4f767f39de7df [id=c9a367ed70c742a18d7dc76e72b37300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.152577 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aee71c51a96482fbb527576867bbaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.159093 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f01daf4c67462ba75e1d0cf56cef12" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:21.159323 1059 raft_consensus.cc:2463] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:21.159416 1770 consensus_queue.cc:1035] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.170886 1443 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 81e26341525d462e9042b1ab06cf7ac3
I20250110 02:51:21.171068 1443 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 28) took 1285 ms. Trace:
I20250110 02:51:21.171168 1443 rpcz_store.cc:276] 0110 02:51:19.885815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.352650 (+466835us) service_pool.cc:225] Handling call
0110 02:51:20.403208 (+ 50558us) spinlock_profiling.cc:243] Waited 50.1 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:20.403605 (+ 397us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.403795 (+ 190us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:20.403871 (+ 76us) catalog_manager.cc:2762] Locking tablets
0110 02:51:20.404004 (+ 133us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:20.481980 (+ 77976us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:20.482084 (+ 104us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:21.170825 (+688741us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:21.171000 (+ 175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.411015 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.411090 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.411095 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:20.411097 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.411204 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.411213 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.411217 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:20.411276 (+ 59us) tablet.cc:698] Row locks acquired
0110 02:51:20.411280 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:20.411305 (+ 25us) write_op.cc:270] Start()
0110 02:51:20.411330 (+ 25us) write_op.cc:276] Timestamp: P: 1736477480411239 usec, L: 0
0110 02:51:20.411334 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.412420 (+ 1086us) log.cc:844] Serialized 533 byte log entry
0110 02:51:20.480100 (+ 67680us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.481615 (+ 1515us) write_op.cc:301] APPLY: starting
0110 02:51:20.481626 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.481659 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.481703 (+ 44us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.481753 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.481785 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:20.481795 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:20.481846 (+ 51us) log.cc:844] Serialized 29 byte log entry
0110 02:51:20.481886 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.481900 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:20.481907 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.481968 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.481971 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":115307392,"child_traces":[["op",{"apply.queue_time_us":1479,"apply.run_cpu_time_us":349,"apply.run_wall_time_us":349,"num_ops":2,"prepare.queue_time_us":6384,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":1532,"raft.queue_time_us":26,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":68747}]]}
I20250110 02:51:21.172232 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_9ea14d3254a34f3cb6825fc2e407b2d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.172896 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea14d3254a34f3cb6825fc2e407b2d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.174407 1439 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:21.174922 1439 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 29) took 1289 ms. Trace:
I20250110 02:51:21.175367 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:21.175040 1439 rpcz_store.cc:276] 0110 02:51:19.885887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.411517 (+525630us) service_pool.cc:225] Handling call
0110 02:51:20.412853 (+ 1336us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.412995 (+ 142us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:20.413061 (+ 66us) catalog_manager.cc:2762] Locking tablets
0110 02:51:20.413182 (+ 121us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:20.504430 (+ 91248us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:20.504541 (+ 111us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:21.174319 (+669778us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:21.174882 (+ 563us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.433631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.433733 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.433738 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:20.433740 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.433851 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.433860 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.433866 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:20.434105 (+ 239us) tablet.cc:698] Row locks acquired
0110 02:51:20.434107 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.434139 (+ 32us) write_op.cc:270] Start()
0110 02:51:20.434165 (+ 26us) write_op.cc:276] Timestamp: P: 1736477480434068 usec, L: 0
0110 02:51:20.434169 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.434359 (+ 190us) log.cc:844] Serialized 756 byte log entry
0110 02:51:20.503741 (+ 69382us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.504009 (+ 268us) write_op.cc:301] APPLY: starting
0110 02:51:20.504024 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.504059 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.504100 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.504164 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.504195 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:20.504212 (+ 17us) write_op.cc:312] APPLY: finished
0110 02:51:20.504271 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:20.504326 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.504341 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:20.504346 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.504417 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.504420 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1940864,"child_traces":[["op",{"apply.queue_time_us":221,"apply.run_cpu_time_us":400,"apply.run_wall_time_us":399,"num_ops":2,"prepare.queue_time_us":19788,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":833,"raft.queue_time_us":1252,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":69543}]]}
I20250110 02:51:21.175917 1439 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_bfbd91df5def489aa8830a6d7345e498" } new_table_name: "default.Table_3993d1a3d35d4f9cb244a585f4d0084e" modify_external_catalogs: true
I20250110 02:51:21.175952 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:21.176709 892 tablet_service.cc:1467] Processing CreateTablet for tablet 89e3bcb739b740b88e98f2717f58e0c2 (DEFAULT_TABLE table=default.Table_829a8c82f1a44c3b9b1b67047764be79 [id=b800401c0e744bd48fda3ee1c851862e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.177138 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e3bcb739b740b88e98f2717f58e0c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:21.177884 1456 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54b538100977456794c9519bbdffa7e7 consensus state has no leader
I20250110 02:51:21.186329 892 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb (DEFAULT_TABLE table=default.Table_bd274cbe850641b69ba5a4b6fa3f2474 [id=378a8ab24eee458b89ddf29fae4d39a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.186830 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c4bf0c381438eafadf4d3d0e8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.192117 891 tablet_service.cc:1467] Processing CreateTablet for tablet 24034fb3da6e47978cdc9b02e714e30d (DEFAULT_TABLE table=default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.192509 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24034fb3da6e47978cdc9b02e714e30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.198434 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.198678 775 raft_consensus.cc:3054] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.200899 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:21.201577 775 raft_consensus.cc:2463] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:21.203197 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820975aa259f4274ba6c19398425e94b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:21.203369 1061 raft_consensus.cc:3054] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.205897 1061 raft_consensus.cc:2463] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:21.206851 886 tablet_service.cc:1467] Processing CreateTablet for tablet 291852e862c24f589d16599a812d0992 (DEFAULT_TABLE table=default.Table_af9287df67924374975e9318755998f4 [id=c60d14c6bea94079a3401319303ccba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.207219 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291852e862c24f589d16599a812d0992. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:21.216938 996 consensus_peers.cc:487] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0127de45264f4660b843734a530eb310. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.219540 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2 (DEFAULT_TABLE table=default.Table_7ac37bfbe9d7409cbd07212eceafa582 [id=81d5e73bc81c4903817b104235cb8eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.219993 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.238646 1770 raft_consensus.cc:491] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.238807 1770 raft_consensus.cc:513] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.239459 1770 leader_election.cc:290] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.240149 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.240350 911 raft_consensus.cc:2463] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.241104 994 leader_election.cc:304] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:21.241528 1770 raft_consensus.cc:2798] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.241604 1770 raft_consensus.cc:491] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.241669 1770 raft_consensus.cc:3054] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.244619 1770 raft_consensus.cc:513] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.245729 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.245896 911 raft_consensus.cc:3054] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.245829 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.246001 774 raft_consensus.cc:3054] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.248896 774 raft_consensus.cc:2463] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.249565 996 leader_election.cc:304] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.249842 1990 raft_consensus.cc:2798] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.249939 1990 raft_consensus.cc:695] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.250190 1990 consensus_queue.cc:237] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.252224 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9018955f3884824beb2e7336987fc1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.252715 775 raft_consensus.cc:2371] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1. Re-sending same reply.
W20250110 02:51:21.254180 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:21.254614 1770 leader_election.cc:290] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:21.257536 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:21.261293 1991 raft_consensus.cc:491] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.261431 1991 raft_consensus.cc:513] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.262120 1991 leader_election.cc:290] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.263028 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.263592 711 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.264746 1770 raft_consensus.cc:491] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.264878 1770 raft_consensus.cc:513] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.265925 1770 leader_election.cc:290] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:21.266597 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.266865 775 raft_consensus.cc:2463] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.267321 996 leader_election.cc:304] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.267926 2055 raft_consensus.cc:2798] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.268002 2055 raft_consensus.cc:491] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.268065 2055 raft_consensus.cc:3054] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.268636 1849 raft_consensus.cc:491] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.268765 1849 raft_consensus.cc:513] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.269387 1849 leader_election.cc:290] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.269960 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.270141 774 raft_consensus.cc:2463] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.270498 996 leader_election.cc:304] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.270725 1849 raft_consensus.cc:2798] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.270792 1849 raft_consensus.cc:491] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.270843 1849 raft_consensus.cc:3054] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.272104 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:21.272603 994 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.274922 2082 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d8d40556dc348e8aa8255d116251a10)
I20250110 02:51:21.275683 1849 raft_consensus.cc:513] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.276253 2055 raft_consensus.cc:513] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.276510 1849 leader_election.cc:290] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.276790 2055 leader_election.cc:290] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:21.277248 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.277426 774 raft_consensus.cc:3054] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.277350 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.277456 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.280417 774 raft_consensus.cc:2463] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:21.280500 994 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:21.280678 994 leader_election.cc:343] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.280812 996 leader_election.cc:304] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:21.281075 2055 raft_consensus.cc:491] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.281212 1849 raft_consensus.cc:2798] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.281188 2055 raft_consensus.cc:513] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.281317 1849 raft_consensus.cc:695] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.281563 1849 consensus_queue.cc:237] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.281970 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.282133 913 raft_consensus.cc:2463] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.282497 994 leader_election.cc:304] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:21.282723 2024 raft_consensus.cc:2798] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.282794 2024 raft_consensus.cc:491] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.282858 2024 raft_consensus.cc:3054] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.283779 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.283941 775 raft_consensus.cc:3054] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.285521 2024 raft_consensus.cc:513] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.286080 2024 leader_election.cc:290] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.286304 2055 leader_election.cc:290] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.286636 775 raft_consensus.cc:2463] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.286811 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.286970 913 raft_consensus.cc:3054] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.287739 996 leader_election.cc:304] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:21.291260 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.291375 2055 raft_consensus.cc:2798] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.291492 2055 raft_consensus.cc:695] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.291728 775 raft_consensus.cc:3054] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.291702 2055 consensus_queue.cc:237] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.293895 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:21.294493 994 leader_election.cc:343] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.291525 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2163c343a694f5ca04ac9f7610d25e3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.298529 775 raft_consensus.cc:2463] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.299072 996 leader_election.cc:304] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.299376 1990 raft_consensus.cc:2798] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.299525 1990 raft_consensus.cc:695] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.299820 1990 consensus_queue.cc:237] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.303871 1188 tablet_bootstrap.cc:492] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.304023 1188 ts_tablet_manager.cc:1397] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.310s user 0.003s sys 0.000s
I20250110 02:51:21.304895 1188 raft_consensus.cc:357] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.305120 1188 raft_consensus.cc:383] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.305181 1188 raft_consensus.cc:738] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.306192 1991 raft_consensus.cc:491] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.306433 1991 raft_consensus.cc:513] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:21.306666 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:21.307041 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3510e86314144a2ca012aacf3d78eacc already in progress: creating tablet
W20250110 02:51:21.307300 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a7397e41f7374e9fb1621dbcf30af703 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7397e41f7374e9fb1621dbcf30af703 already in progress: creating tablet
I20250110 02:51:21.307482 1991 leader_election.cc:290] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.309224 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.309756 711 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:21.310500 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
I20250110 02:51:21.315208 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e (DEFAULT_TABLE table=default.Table_4ef1477d7e314ddbaa6fd875538bc552 [id=d7c9b51aed334036850b6f60bbcb4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:21.315438 709 leader_election.cc:343] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.315537 709 leader_election.cc:304] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.315589 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a444bfd73104e9f96e1aaae7ce9ce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.315922 1991 raft_consensus.cc:2743] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:21.317059 853 consensus_peers.cc:487] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d4422eca5604d149e73c2a669d07af0. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.318770 898 tablet_service.cc:1467] Processing CreateTablet for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (DEFAULT_TABLE table=default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.319062 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd02e9f8cd0748a0a6a3202fbe390ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.321919 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:21.322213 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:21.322340 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:21.322477 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:21.327191 853 consensus_peers.cc:487] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60e534bf66f044b88d8cda0f3a1228f4. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:21.327436 853 leader_election.cc:343] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:21.327637 853 leader_election.cc:304] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.332850 911 raft_consensus.cc:2463] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.336750 913 raft_consensus.cc:2463] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.275811 2082 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:21.350824 2075 tablet_replica.cc:331] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:21.351231 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.351943 1990 consensus_queue.cc:1035] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.353725 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.354498 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:21.355264 709 leader_election.cc:343] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.355854 709 leader_election.cc:304] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:21.356276 709 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.356324 1832 raft_consensus.cc:2798] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.356370 709 leader_election.cc:304] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.356436 1832 raft_consensus.cc:695] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:21.356680 1991 raft_consensus.cc:2743] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.356658 1832 consensus_queue.cc:237] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.356817 1991 raft_consensus.cc:2743] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:21.357349 709 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.357462 709 leader_election.cc:304] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.357775 1991 raft_consensus.cc:2743] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.366537 2075 raft_consensus.cc:2238] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:21.366725 2075 raft_consensus.cc:2267] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:21.367250 2075 ts_tablet_manager.cc:1905] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:21.385362 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 57714.6us
I20250110 02:51:21.387060 2082 leader_election.cc:290] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
W20250110 02:51:21.387929 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54b538100977456794c9519bbdffa7e7 consensus state has no leader
I20250110 02:51:21.391471 1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fca9a92f47145fcaaf4f65038d6f558" candidate_term: 5 candidate_status { last_received { term: 4 index: 680 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:21.392622 1457 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 20) took 1507 ms. Trace:
I20250110 02:51:21.392769 1457 rpcz_store.cc:276] 0110 02:51:19.885010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.121136 (+236126us) service_pool.cc:225] Handling call
0110 02:51:20.291454 (+170318us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:20.291668 (+ 214us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.291845 (+ 177us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:20.291968 (+ 123us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:20.292032 (+ 64us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:20.421604 (+129572us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:21.392560 (+970956us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.406021 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.406126 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.406141 (+ 15us) write_op.cc:435] Acquired schema lock
0110 02:51:20.406144 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:20.406244 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.406252 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.406256 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:20.406308 (+ 52us) tablet.cc:698] Row locks acquired
0110 02:51:20.406310 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.406335 (+ 25us) write_op.cc:270] Start()
0110 02:51:20.406359 (+ 24us) write_op.cc:276] Timestamp: P: 1736477480406269 usec, L: 0
0110 02:51:20.406362 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.406543 (+ 181us) log.cc:844] Serialized 405 byte log entry
0110 02:51:20.419471 (+ 12928us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.419713 (+ 242us) write_op.cc:301] APPLY: starting
0110 02:51:20.419727 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.419760 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.419793 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.419835 (+ 42us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.419860 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:20.419866 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.419925 (+ 59us) log.cc:844] Serialized 23 byte log entry
0110 02:51:20.419995 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.420011 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:20.421439 (+ 1428us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.423966 (+ 2527us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x616000051178. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f9532502c81 00007f95324fb136 00007f95324f93a8 00007f95324fc420 00007f95324fc22d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:20.424011 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.424016 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":391205632,"child_traces":[["op",{"apply.queue_time_us":205,"apply.run_cpu_time_us":757,"apply.run_wall_time_us":4280,"num_ops":1,"prepare.queue_time_us":113645,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":648,"raft.queue_time_us":33,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":13084,"spinlock_wait_cycles":4142464}]]}
I20250110 02:51:21.400727 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_d3ddbdeb213d4d65a47a41e1912df72d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.401903 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ddbdeb213d4d65a47a41e1912df72d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:21.402665 442 leader_election.cc:336] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449): Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111)
I20250110 02:51:21.403084 442 leader_election.cc:304] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fca9a92f47145fcaaf4f65038d6f558; no voters: 5d8d40556dc348e8aa8255d116251a10, 8bc934cc708045e9858ee8f8e64660cd
I20250110 02:51:21.403199 1441 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 22) took 1517 ms. Trace:
I20250110 02:51:21.403281 1441 rpcz_store.cc:276] 0110 02:51:19.885245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.271251 (+386006us) service_pool.cc:225] Handling call
0110 02:51:20.286754 (+ 15503us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:20.287015 (+ 261us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:20.287194 (+ 179us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:20.287379 (+ 185us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:20.287438 (+ 59us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:20.420888 (+133450us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:21.403155 (+982267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.407245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.407348 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.407352 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:20.407354 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:20.407438 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.407448 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.407456 (+ 8us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:20.407500 (+ 44us) tablet.cc:698] Row locks acquired
0110 02:51:20.407502 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.407525 (+ 23us) write_op.cc:270] Start()
0110 02:51:20.407547 (+ 22us) write_op.cc:276] Timestamp: P: 1736477480407460 usec, L: 0
0110 02:51:20.407550 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.407664 (+ 114us) log.cc:844] Serialized 405 byte log entry
0110 02:51:20.419568 (+ 11904us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.420539 (+ 971us) write_op.cc:301] APPLY: starting
0110 02:51:20.420551 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.420591 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.420621 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.420657 (+ 36us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.420689 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:20.420695 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.420746 (+ 51us) log.cc:844] Serialized 23 byte log entry
0110 02:51:20.420779 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.420792 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:20.420797 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:20.420863 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.420867 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":34903936,"child_traces":[["op",{"apply.queue_time_us":927,"apply.run_cpu_time_us":321,"apply.run_wall_time_us":320,"mutex_wait_us":8440,"num_ops":1,"prepare.queue_time_us":119441,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":540,"raft.queue_time_us":35,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":12006}]]}
I20250110 02:51:21.404245 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_1fbfd53f85064ec8b3dd74e0a4d033c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.404716 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fbfd53f85064ec8b3dd74e0a4d033c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.405686 2082 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8fca9a92f47145fcaaf4f65038d6f558 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 02:51:21.305374 1188 consensus_queue.cc:260] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:21.411288 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
W20250110 02:51:21.411808 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:21.412259 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3510e86314144a2ca012aacf3d78eacc already in progress: creating tablet
W20250110 02:51:21.412690 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a7397e41f7374e9fb1621dbcf30af703 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7397e41f7374e9fb1621dbcf30af703 already in progress: creating tablet
I20250110 02:51:21.415266 1184 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.415405 1184 ts_tablet_manager.cc:1397] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.272s user 0.029s sys 0.060s
I20250110 02:51:21.416177 1184 raft_consensus.cc:357] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.416378 1184 raft_consensus.cc:383] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.416450 1184 raft_consensus.cc:738] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.416695 1184 consensus_queue.cc:260] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.417968 1184 ts_tablet_manager.cc:1428] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:21.420434 853 leader_election.cc:304] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:21.420934 1832 raft_consensus.cc:2798] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.421028 1832 raft_consensus.cc:491] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.421090 1832 raft_consensus.cc:3054] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.422821 1181 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.422974 1181 ts_tablet_manager.cc:1397] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.296s user 0.038s sys 0.065s
I20250110 02:51:21.423789 1181 raft_consensus.cc:357] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.423979 1181 raft_consensus.cc:383] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.424036 1181 raft_consensus.cc:738] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.424727 1181 consensus_queue.cc:260] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.424394 1832 raft_consensus.cc:513] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.425390 1181 ts_tablet_manager.cc:1428] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.425866 1832 leader_election.cc:290] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.426882 1849 consensus_queue.cc:1035] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.427810 1181 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.428143 1184 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.430575 1184 tablet_bootstrap.cc:654] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.430615 1181 tablet_bootstrap.cc:654] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.443800 1181 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.443935 1181 ts_tablet_manager.cc:1397] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20250110 02:51:21.444574 1181 raft_consensus.cc:357] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.444759 1181 raft_consensus.cc:383] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.444810 1181 raft_consensus.cc:738] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.446015 1188 ts_tablet_manager.cc:1428] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.142s user 0.002s sys 0.001s
I20250110 02:51:21.445044 1181 consensus_queue.cc:260] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.448020 1181 ts_tablet_manager.cc:1428] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:21.447889 1188 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.449944 1181 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.450974 2092 raft_consensus.cc:491] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.451095 2092 raft_consensus.cc:513] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.451329 1188 tablet_bootstrap.cc:654] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.451633 2092 leader_election.cc:290] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.452093 1184 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.452212 1184 ts_tablet_manager.cc:1397] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20250110 02:51:21.452558 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:21.452905 1184 raft_consensus.cc:357] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:21.453071 711 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.453384 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.456971 1181 tablet_bootstrap.cc:654] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:21.457212 709 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.457310 709 leader_election.cc:304] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.457712 2092 raft_consensus.cc:2743] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.453075 1184 raft_consensus.cc:383] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.457938 1184 raft_consensus.cc:738] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.458254 1184 consensus_queue.cc:260] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.458896 2075 ts_tablet_manager.cc:1918] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:21.459007 1916 consensus_queue.cc:1035] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.462764 1916 consensus_queue.cc:1035] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:21.476449 853 consensus_peers.cc:487] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.479557 1832 consensus_queue.cc:1035] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.480739 1832 consensus_queue.cc:1035] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:21.498699 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 103302us
I20250110 02:51:21.505363 2075 log.cc:1198] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:21.459472 1184 ts_tablet_manager.cc:1428] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250110 02:51:21.505815 1184 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.508028 1184 tablet_bootstrap.cc:654] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.512612 2075 ts_tablet_manager.cc:1939] T 3510e86314144a2ca012aacf3d78eacc P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:21.514523 1188 tablet_bootstrap.cc:492] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.514498 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 3510e86314144a2ca012aacf3d78eacc (table default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]) successfully deleted
I20250110 02:51:21.514649 1188 ts_tablet_manager.cc:1397] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.003s
I20250110 02:51:21.515810 1188 raft_consensus.cc:357] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.516008 1188 raft_consensus.cc:383] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.516058 1188 raft_consensus.cc:738] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.516290 1188 consensus_queue.cc:260] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.517726 1188 ts_tablet_manager.cc:1428] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:21.518777 1188 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:21.522608 853 consensus_peers.cc:487] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.523558 1957 consensus_queue.cc:1035] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.524004 1957 consensus_queue.cc:1035] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.524873 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f01daf4c67462ba75e1d0cf56cef12" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:21.525787 1061 raft_consensus.cc:3054] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.528074 2075 tablet_replica.cc:331] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:21.528326 2075 raft_consensus.cc:2238] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:21.528513 2075 raft_consensus.cc:2267] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:21.529095 2075 ts_tablet_manager.cc:1905] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:21.529330 1957 consensus_queue.cc:1035] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:21.529747 853 leader_election.cc:343] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:21.536111 895 tablet_service.cc:1467] Processing CreateTablet for tablet bcd0b68aa9134635875b453889718cb4 (DEFAULT_TABLE table=default.Table_b0925c3e58414c7a9582b4ede6d7b5b4 [id=c159709e198b4bdbbf72381cbba92eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.536615 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd0b68aa9134635875b453889718cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.546682 1957 consensus_queue.cc:1035] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.548902 1916 consensus_queue.cc:1035] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.550637 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bcd0b68aa9134635875b453889718cb4 (DEFAULT_TABLE table=default.Table_b0925c3e58414c7a9582b4ede6d7b5b4 [id=c159709e198b4bdbbf72381cbba92eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.551154 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd0b68aa9134635875b453889718cb4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:21.570878 853 consensus_peers.cc:487] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.533247 743 tablet_service.cc:1467] Processing CreateTablet for tablet bcd0b68aa9134635875b453889718cb4 (DEFAULT_TABLE table=default.Table_b0925c3e58414c7a9582b4ede6d7b5b4 [id=c159709e198b4bdbbf72381cbba92eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:21.576321 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd0b68aa9134635875b453889718cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.570084 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
W20250110 02:51:21.582135 853 consensus_peers.cc:487] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.582525 1916 consensus_queue.cc:1035] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.587591 1188 tablet_bootstrap.cc:654] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.593541 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 40) took 1114 ms. Trace:
I20250110 02:51:21.593678 1448 rpcz_store.cc:276] 0110 02:51:20.478884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.140733 (+661849us) service_pool.cc:225] Handling call
0110 02:51:21.391367 (+250634us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.392073 (+ 706us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.593279 (+201206us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.593399 (+ 120us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.593486 (+ 87us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.451983 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.452086 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.452093 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:21.452097 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:21.452189 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.452199 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.452205 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.452272 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:21.452274 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.452303 (+ 29us) write_op.cc:270] Start()
0110 02:51:21.452331 (+ 28us) write_op.cc:276] Timestamp: P: 1736477481452248 usec, L: 0
0110 02:51:21.452334 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.452504 (+ 170us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.585872 (+133368us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.592491 (+ 6619us) write_op.cc:301] APPLY: starting
0110 02:51:21.592506 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.592542 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.592593 (+ 51us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.592714 (+ 121us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.592743 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.592750 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.592822 (+ 72us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.593180 (+ 358us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.593197 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:21.593203 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6522,"mutex_wait_us":1,"num_ops":2,"prepare.queue_time_us":59479,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":932,"raft.queue_time_us":31,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":133503}]]}
I20250110 02:51:21.595685 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 42) took 1107 ms. Trace:
I20250110 02:51:21.595786 1443 rpcz_store.cc:276] 0110 02:51:20.488265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.171981 (+683716us) service_pool.cc:225] Handling call
0110 02:51:21.398410 (+226429us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.398899 (+ 489us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.595468 (+196569us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.595563 (+ 95us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.595629 (+ 66us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.452942 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.453024 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.453030 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:21.453032 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.453117 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.453125 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.453129 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.453191 (+ 62us) tablet.cc:698] Row locks acquired
0110 02:51:21.453193 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.453219 (+ 26us) write_op.cc:270] Start()
0110 02:51:21.453241 (+ 22us) write_op.cc:276] Timestamp: P: 1736477481453165 usec, L: 0
0110 02:51:21.453244 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.453366 (+ 122us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.585956 (+132590us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.594638 (+ 8682us) write_op.cc:301] APPLY: starting
0110 02:51:21.594650 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.594695 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.594720 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.594825 (+ 105us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.594855 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.594860 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:21.594919 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.595221 (+ 302us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.595234 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:21.595240 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8647,"num_ops":2,"prepare.queue_time_us":53558,"prepare.run_cpu_time_us":501,"prepare.run_wall_time_us":769,"raft.queue_time_us":28,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":132695}]]}
I20250110 02:51:21.598322 1061 raft_consensus.cc:2463] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:21.599010 853 leader_election.cc:304] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:21.599319 1916 raft_consensus.cc:2798] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.599514 1916 raft_consensus.cc:695] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:21.599860 1916 consensus_queue.cc:237] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.606673 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_4f4e7744b0f5403891cb46da79efba4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.607276 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f4e7744b0f5403891cb46da79efba4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.608340 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_dab0925165d74fb087cc0f47cca63c92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.608893 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dab0925165d74fb087cc0f47cca63c92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.618561 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 44) took 1050 ms. Trace:
I20250110 02:51:21.618733 1456 rpcz_store.cc:276] 0110 02:51:20.567681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.400530 (+832849us) service_pool.cc:225] Handling call
0110 02:51:21.402112 (+ 1582us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.402700 (+ 588us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.618293 (+215593us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.618442 (+ 149us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.618528 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.454566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.454639 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.454643 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:21.454645 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.454708 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.454715 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.454720 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.454773 (+ 53us) tablet.cc:698] Row locks acquired
0110 02:51:21.454777 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:21.454804 (+ 27us) write_op.cc:270] Start()
0110 02:51:21.454828 (+ 24us) write_op.cc:276] Timestamp: P: 1736477481454748 usec, L: 0
0110 02:51:21.454832 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.454952 (+ 120us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.605957 (+151005us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.617528 (+ 11571us) write_op.cc:301] APPLY: starting
0110 02:51:21.617544 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.617584 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.617637 (+ 53us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.617756 (+ 119us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.617789 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.617796 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.617874 (+ 78us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.618192 (+ 318us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.618210 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:21.618217 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11490,"num_ops":2,"prepare.queue_time_us":51492,"prepare.run_cpu_time_us":520,"prepare.run_wall_time_us":38632,"raft.queue_time_us":35,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":151094}]]}
I20250110 02:51:21.619527 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_610281233be84be48b3fd966dbbcdec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.620093 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_610281233be84be48b3fd966dbbcdec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.622578 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 45) took 1007 ms. Trace:
I20250110 02:51:21.622668 1441 rpcz_store.cc:276] 0110 02:51:20.614796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.404029 (+789233us) service_pool.cc:225] Handling call
0110 02:51:21.404812 (+ 783us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.405223 (+ 411us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.622339 (+217116us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.622462 (+ 123us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.622531 (+ 69us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.493232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.493359 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.493366 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:21.493368 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.493467 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.493478 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.493483 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.493564 (+ 81us) tablet.cc:698] Row locks acquired
0110 02:51:21.493566 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.493600 (+ 34us) write_op.cc:270] Start()
0110 02:51:21.493635 (+ 35us) write_op.cc:276] Timestamp: P: 1736477481493540 usec, L: 0
0110 02:51:21.493638 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.493828 (+ 190us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.621473 (+127645us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.621773 (+ 300us) write_op.cc:301] APPLY: starting
0110 02:51:21.621787 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.621822 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.621849 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.621946 (+ 97us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.621990 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.621997 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.622066 (+ 69us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.622252 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.622266 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:21.622272 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":230,"num_ops":2,"prepare.queue_time_us":87644,"prepare.run_cpu_time_us":700,"prepare.run_wall_time_us":1202,"raft.queue_time_us":33,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":127806}]]}
I20250110 02:51:21.624004 1188 tablet_bootstrap.cc:492] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.624142 1188 ts_tablet_manager.cc:1397] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.106s user 0.002s sys 0.002s
I20250110 02:51:21.632776 1181 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.632974 1181 ts_tablet_manager.cc:1397] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.184s user 0.017s sys 0.016s
I20250110 02:51:21.633484 1188 raft_consensus.cc:357] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.633740 1188 raft_consensus.cc:383] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.633800 1188 raft_consensus.cc:738] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.633955 1181 raft_consensus.cc:357] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.634217 1181 raft_consensus.cc:383] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.634284 1181 raft_consensus.cc:738] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.634552 1188 consensus_queue.cc:260] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.635195 1188 ts_tablet_manager.cc:1428] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250110 02:51:21.647015 1188 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.651670 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_eeee9b94a4844bbdb5ced2be8c6cdaf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.652208 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeee9b94a4844bbdb5ced2be8c6cdaf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.654699 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 47) took 1027 ms. Trace:
I20250110 02:51:21.654803 1443 rpcz_store.cc:276] 0110 02:51:20.626832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.597306 (+970474us) service_pool.cc:225] Handling call
0110 02:51:21.606590 (+ 9284us) spinlock_profiling.cc:243] Waited 9.02 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfbd03 00007f952fefe236 00007f952fefdf65 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:21.607371 (+ 781us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.607819 (+ 448us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.654446 (+ 46627us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.654555 (+ 109us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.654645 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.610002 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.610134 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.610142 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:21.610146 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:21.610253 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.610267 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.610274 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.610366 (+ 92us) tablet.cc:698] Row locks acquired
0110 02:51:21.610368 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.610425 (+ 57us) write_op.cc:270] Start()
0110 02:51:21.610451 (+ 26us) write_op.cc:276] Timestamp: P: 1736477481610411 usec, L: 0
0110 02:51:21.610453 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.610640 (+ 187us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.653575 (+ 42935us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.653735 (+ 160us) write_op.cc:301] APPLY: starting
0110 02:51:21.653751 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.653796 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.653824 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.653916 (+ 92us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.653947 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.653953 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:21.654053 (+ 100us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.654318 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.654335 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:21.654341 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":20742528,"child_traces":[["op",{"apply.queue_time_us":115,"num_ops":2,"prepare.queue_time_us":1665,"prepare.run_cpu_time_us":5605,"prepare.run_wall_time_us":40686,"raft.queue_time_us":3292,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"replication_time_us":43098,"thread_start_us":39502,"threads_started":1}]]}
I20250110 02:51:21.655265 912 raft_consensus.cc:3054] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.655582 1188 tablet_bootstrap.cc:654] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.655620 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_67bc8c66612446d8a0702142479b305e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.656111 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67bc8c66612446d8a0702142479b305e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.659897 1849 consensus_queue.cc:1035] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.668509 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 46) took 1044 ms. Trace:
I20250110 02:51:21.668661 1448 rpcz_store.cc:276] 0110 02:51:20.624420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.594299 (+969879us) service_pool.cc:225] Handling call
0110 02:51:21.606279 (+ 11980us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfbd03 00007f952fefe236 00007f952fefdf65 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:21.609006 (+ 2727us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:21.609487 (+ 481us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:21.668241 (+ 58754us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:21.668382 (+ 141us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:21.668470 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.650716 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.650833 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.650840 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:21.650845 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:21.650944 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.650956 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.650961 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.651071 (+ 110us) tablet.cc:698] Row locks acquired
0110 02:51:21.651073 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.651104 (+ 31us) write_op.cc:270] Start()
0110 02:51:21.651136 (+ 32us) write_op.cc:276] Timestamp: P: 1736477481651087 usec, L: 0
0110 02:51:21.651140 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.661148 (+ 10008us) log.cc:844] Serialized 433 byte log entry
0110 02:51:21.661720 (+ 572us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:21.667018 (+ 5298us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.667616 (+ 598us) write_op.cc:301] APPLY: starting
0110 02:51:21.667633 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.667677 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.667680 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.667803 (+ 123us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.667838 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:21.667845 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.667914 (+ 69us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.668144 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.668161 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:21.668167 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":26814848,"child_traces":[["op",{"apply.queue_time_us":531,"num_ops":2,"prepare.queue_time_us":40771,"prepare.run_cpu_time_us":983,"prepare.run_wall_time_us":11061,"raft.queue_time_us":1049,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":283,"replication_time_us":15851,"spinlock_wait_cycles":22570240}]]}
I20250110 02:51:21.671455 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 76833ce515cf4ee9bc8fff0f626af220 from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:21.674986 1434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 -> Peer 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449): Couldn't send request to peer 8bc934cc708045e9858ee8f8e64660cd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250110 02:51:21.677968 2075 ts_tablet_manager.cc:1918] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:21.678720 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:21.679426 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:21.682235 2075 log.cc:1198] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:21.682677 2075 ts_tablet_manager.cc:1939] T a7397e41f7374e9fb1621dbcf30af703 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:21.702803 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a7397e41f7374e9fb1621dbcf30af703 (table default.Table_94cf6a20190647dd87d3616b069b0243 [id=a91e19e65fd74454b4b4d339a5cbe806]) successfully deleted
I20250110 02:51:21.708700 1441 master_service.cc:946] ReplaceTablet: received request to replace tablet 95b14d7592024a34baf7756d92e546d9 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:21.711022 1680 consensus_queue.cc:1035] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.714253 1184 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.716481 1184 ts_tablet_manager.cc:1397] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.009s
I20250110 02:51:21.717226 1184 raft_consensus.cc:357] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.717411 1184 raft_consensus.cc:383] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.717475 1184 raft_consensus.cc:738] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.717679 1184 consensus_queue.cc:260] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.718812 1184 ts_tablet_manager.cc:1428] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.720175 1188 tablet_bootstrap.cc:492] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.720309 1188 ts_tablet_manager.cc:1397] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20250110 02:51:21.721587 1188 raft_consensus.cc:357] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.721801 1188 raft_consensus.cc:383] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.721854 1188 raft_consensus.cc:738] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.722069 1188 consensus_queue.cc:260] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.722563 1188 ts_tablet_manager.cc:1428] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:21.723875 1184 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.724195 1188 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.725769 1184 tablet_bootstrap.cc:654] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.725870 1188 tablet_bootstrap.cc:654] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.739270 1181 consensus_queue.cc:260] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.741984 2092 raft_consensus.cc:491] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.742172 2092 raft_consensus.cc:513] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.742980 2092 leader_election.cc:290] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.743247 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:21.743747 709 leader_election.cc:343] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.745218 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.745862 711 leader_election.cc:343] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.745986 711 leader_election.cc:304] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.746546 2092 raft_consensus.cc:2743] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.751595 1448 master_service.cc:946] ReplaceTablet: received request to replace tablet 82bd4307e5fe4f61b9308005cf3446e1 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:21.752954 1181 ts_tablet_manager.cc:1428] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.120s user 0.003s sys 0.000s
I20250110 02:51:21.753065 1188 tablet_bootstrap.cc:492] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.753225 1188 ts_tablet_manager.cc:1397] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20250110 02:51:21.754105 1181 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.753983 1188 raft_consensus.cc:357] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.754262 1188 raft_consensus.cc:383] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.754328 1188 raft_consensus.cc:738] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.754559 1188 consensus_queue.cc:260] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.755048 1188 ts_tablet_manager.cc:1428] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.755281 1188 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.757622 1181 tablet_bootstrap.cc:654] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.757639 1188 tablet_bootstrap.cc:654] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.762398 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e392c98c33b4cbbb45dd0b4ec6b7ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:21.768491 1680 raft_consensus.cc:491] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.768622 1680 raft_consensus.cc:513] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.769335 1680 leader_election.cc:290] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.769436 1184 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.769584 1184 ts_tablet_manager.cc:1397] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250110 02:51:21.770051 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.770222 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.770418 775 raft_consensus.cc:2463] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:21.770730 994 leader_election.cc:343] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.770918 996 leader_election.cc:304] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:21.771284 1680 raft_consensus.cc:2798] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.771358 1680 raft_consensus.cc:491] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.771422 1680 raft_consensus.cc:3054] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.771481 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:21.771714 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:21.771875 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:21.772133 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:21.772281 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:21.772439 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:21.772655 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:21.772818 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:21.772763 1184 raft_consensus.cc:357] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.773015 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:21.773293 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:21.773423 1184 raft_consensus.cc:383] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.773432 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:21.773543 1184 raft_consensus.cc:738] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.773921 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:21.773828 1680 raft_consensus.cc:513] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.774119 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:21.774302 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:21.774295 1184 consensus_queue.cc:260] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.774726 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.774838 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:21.774874 1680 leader_election.cc:290] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.775002 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:21.775233 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:21.774744 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0588e5e5714cc5bd1c73665e5645b1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:21.775279 994 leader_election.cc:343] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.775418 775 raft_consensus.cc:3054] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.775429 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:21.777947 775 raft_consensus.cc:2463] T 6e0588e5e5714cc5bd1c73665e5645b1 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.778546 996 leader_election.cc:304] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:21.779208 1680 raft_consensus.cc:2798] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.779325 1680 raft_consensus.cc:695] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.779531 1680 consensus_queue.cc:237] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.788097 1184 ts_tablet_manager.cc:1428] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250110 02:51:21.788559 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:21.788863 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:21.789479 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:21.789780 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:21.790423 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 1 got 0
I20250110 02:51:21.790603 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:21.790932 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:21.791092 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:21.791338 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:21.791832 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:21.792104 1444 catalog_manager.cc:5526] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:21.792953 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:21.793143 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:21.793462 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:21.793912 1444 catalog_manager.cc:5526] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:21.794601 1184 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.796309 1184 tablet_bootstrap.cc:654] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.797869 1188 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.798048 1188 ts_tablet_manager.cc:1397] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20250110 02:51:21.798203 2092 raft_consensus.cc:491] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.798326 2092 raft_consensus.cc:513] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.798883 1188 raft_consensus.cc:357] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.799005 2092 leader_election.cc:290] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.799063 1188 raft_consensus.cc:383] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.799465 1188 raft_consensus.cc:738] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.799770 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.800072 1188 consensus_queue.cc:260] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:21.800237 709 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.800360 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.800812 711 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.800911 711 leader_election.cc:304] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:21.801208 2092 raft_consensus.cc:2743] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:21.803679 1188 ts_tablet_manager.cc:1428] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:21.803977 1188 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.810611 1181 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.810741 1181 ts_tablet_manager.cc:1397] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.000s
I20250110 02:51:21.810814 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:21.811036 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:21.811195 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:21.811407 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:21.810891 1188 tablet_bootstrap.cc:654] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.811666 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:21.811705 1181 raft_consensus.cc:357] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.811882 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:21.811923 1181 raft_consensus.cc:383] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.811977 1181 raft_consensus.cc:738] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.812422 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:21.812629 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:21.812811 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:21.813613 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:21.813983 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:21.814203 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:21.814543 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3352f8378b9846a3854086293f2d4e15 (table default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee]). Expected version 1 got 0
I20250110 02:51:21.814704 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:21.814853 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:21.814978 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:21.815253 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:21.816989 1181 consensus_queue.cc:260] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.819864 1181 ts_tablet_manager.cc:1428] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250110 02:51:21.824005 1184 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.824165 1184 ts_tablet_manager.cc:1397] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20250110 02:51:21.824771 1181 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.826468 1181 tablet_bootstrap.cc:654] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.830881 1184 raft_consensus.cc:357] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.831457 1184 raft_consensus.cc:383] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.831519 1184 raft_consensus.cc:738] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.831722 1184 consensus_queue.cc:260] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.836731 1181 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.836851 1181 ts_tablet_manager.cc:1397] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250110 02:51:21.837647 1181 raft_consensus.cc:357] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.837826 1181 raft_consensus.cc:383] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.837874 1181 raft_consensus.cc:738] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.838096 1181 consensus_queue.cc:260] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.839550 1181 ts_tablet_manager.cc:1428] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:21.840555 1181 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.841981 1184 ts_tablet_manager.cc:1428] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250110 02:51:21.844378 1181 tablet_bootstrap.cc:654] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.845026 1184 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.846949 1680 raft_consensus.cc:491] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.847069 1680 raft_consensus.cc:513] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.847671 1680 leader_election.cc:290] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.848161 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.848327 775 raft_consensus.cc:2463] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.848671 996 leader_election.cc:304] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.848886 1680 raft_consensus.cc:2798] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.848938 1680 raft_consensus.cc:491] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.848989 1680 raft_consensus.cc:3054] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.849476 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.849654 914 raft_consensus.cc:2463] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.850638 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 52) took 1012 ms. Trace:
I20250110 02:51:21.850755 1456 rpcz_store.cc:276] 0110 02:51:20.838500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.671380 (+832880us) service_pool.cc:225] Handling call
0110 02:51:21.680718 (+ 9338us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:21.850572 (+169854us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20692352}
I20250110 02:51:21.851186 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet d761ef0d1e5542b1979c29bb2d579489 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:21.851315 1680 raft_consensus.cc:513] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.852097 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.852236 912 raft_consensus.cc:3054] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.853334 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6df074974974d81a83420e7a3a78070" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.853474 775 raft_consensus.cc:3054] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.854630 912 raft_consensus.cc:2463] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.855158 994 leader_election.cc:304] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:21.855463 1990 raft_consensus.cc:2798] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.855573 1990 raft_consensus.cc:695] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.855815 775 raft_consensus.cc:2463] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.855777 1990 consensus_queue.cc:237] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.856087 1188 tablet_bootstrap.cc:492] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.856179 1680 leader_election.cc:290] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.856199 1188 ts_tablet_manager.cc:1397] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.003s
I20250110 02:51:21.856469 2038 consensus_queue.cc:1035] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.857074 1188 raft_consensus.cc:357] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.857283 1188 raft_consensus.cc:383] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.857344 1188 raft_consensus.cc:738] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.857627 1188 consensus_queue.cc:260] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.858064 1184 tablet_bootstrap.cc:654] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.858422 1188 ts_tablet_manager.cc:1428] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:21.858752 1188 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.864840 1441 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_97ca49cb31634f579d9d84e0d4fc8057" } modify_external_catalogs: true
I20250110 02:51:21.865038 1441 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_97ca49cb31634f579d9d84e0d4fc8057" } modify_external_catalogs: true
I20250110 02:51:21.868330 1188 tablet_bootstrap.cc:654] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.868403 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 82bd4307e5fe4f61b9308005cf3446e1
I20250110 02:51:21.868822 1181 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.868885 1448 catalog_manager.cc:6596] ReplaceTablet: tablet 82bd4307e5fe4f61b9308005cf3446e1 deleted and replaced by tablet 4f015e99f53f48599ac2d7a29220bc39
I20250110 02:51:21.868943 1181 ts_tablet_manager.cc:1397] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.003s
I20250110 02:51:21.869050 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:21.869180 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_051f5b2057e84287934aac189c8ea3da" } modify_external_catalogs: true
I20250110 02:51:21.869324 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_051f5b2057e84287934aac189c8ea3da" } modify_external_catalogs: true
I20250110 02:51:21.869889 1181 raft_consensus.cc:357] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.870067 1181 raft_consensus.cc:383] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.870116 1181 raft_consensus.cc:738] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.870299 1181 consensus_queue.cc:260] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:21.870764 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
I20250110 02:51:21.870939 1181 ts_tablet_manager.cc:1428] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:21.871433 1181 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.873505 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:21.873677 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:21.874748 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
I20250110 02:51:21.876245 1456 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_6162127f127347419432d61cdac426bf" } modify_external_catalogs: true
I20250110 02:51:21.876408 1456 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_6162127f127347419432d61cdac426bf" } modify_external_catalogs: true
I20250110 02:51:21.876434 2038 raft_consensus.cc:491] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.876539 2038 raft_consensus.cc:513] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.877426 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.877667 775 raft_consensus.cc:2463] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:21.878036 853 leader_election.cc:304] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.878324 2138 raft_consensus.cc:2798] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.878458 2138 raft_consensus.cc:491] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.878540 2138 raft_consensus.cc:3054] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.878887 2038 leader_election.cc:290] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.878948 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:21.879407 853 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.881078 1181 tablet_bootstrap.cc:654] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.881371 2138 raft_consensus.cc:513] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.884042 2138 leader_election.cc:290] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:21.884585 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:21.885270 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43bb790e85343308a1bb1e1110bfb9c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.885464 775 raft_consensus.cc:3054] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.888350 775 raft_consensus.cc:2463] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:21.888779 853 leader_election.cc:304] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:21.889094 853 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:21.889498 2038 raft_consensus.cc:2798] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.889614 2038 raft_consensus.cc:695] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:21.889838 2038 consensus_queue.cc:237] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.890530 2055 consensus_queue.cc:1035] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:21.890794 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
W20250110 02:51:21.892385 853 consensus_peers.cc:487] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3ae68e4f0254ef38aafd5a1f8f88a54. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:21.894026 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 02:51:21.894512 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 02:51:21.894665 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250110 02:51:21.897351 1188 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.897580 1184 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.897784 1184 ts_tablet_manager.cc:1397] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.005s
I20250110 02:51:21.898046 1188 ts_tablet_manager.cc:1397] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.002s
I20250110 02:51:21.898741 1184 raft_consensus.cc:357] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.898918 1184 raft_consensus.cc:383] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.898973 1184 raft_consensus.cc:738] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:21.899050 996 consensus_peers.cc:487] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.899257 1184 consensus_queue.cc:260] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.900193 1184 ts_tablet_manager.cc:1428] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.900344 1188 raft_consensus.cc:357] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.900764 1188 raft_consensus.cc:383] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.900959 1188 raft_consensus.cc:738] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.900996 1181 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.901134 1181 ts_tablet_manager.cc:1397] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250110 02:51:21.901365 1188 consensus_queue.cc:260] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.901763 1181 raft_consensus.cc:357] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.901937 1181 raft_consensus.cc:383] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.901986 1181 raft_consensus.cc:738] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.902199 1181 consensus_queue.cc:260] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.902824 1188 ts_tablet_manager.cc:1428] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:21.903141 1188 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.903524 1181 ts_tablet_manager.cc:1428] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:21.905371 1184 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.911298 1181 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.912863 1181 tablet_bootstrap.cc:654] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.913146 1680 raft_consensus.cc:491] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.913264 1680 raft_consensus.cc:513] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.913818 1680 leader_election.cc:290] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.914265 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.914474 775 raft_consensus.cc:2463] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.914996 1188 tablet_bootstrap.cc:654] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.915040 996 leader_election.cc:304] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.915437 1680 raft_consensus.cc:2798] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.915506 1680 raft_consensus.cc:491] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.915560 1680 raft_consensus.cc:3054] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.917929 1680 raft_consensus.cc:513] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.919023 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.919170 775 raft_consensus.cc:3054] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.921536 775 raft_consensus.cc:2463] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.921926 996 leader_election.cc:304] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.922133 1849 raft_consensus.cc:2798] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.922210 1849 raft_consensus.cc:695] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.922444 1849 consensus_queue.cc:237] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.924715 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.924942 914 raft_consensus.cc:3054] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.924712 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6de957fd14af983fc1a3cd8629226" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.925606 1680 leader_election.cc:290] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.926478 1184 tablet_bootstrap.cc:654] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.927829 914 raft_consensus.cc:2463] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.931322 1188 tablet_bootstrap.cc:492] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.931525 1188 ts_tablet_manager.cc:1397] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20250110 02:51:21.932222 1188 raft_consensus.cc:357] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.932430 1188 raft_consensus.cc:383] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.932510 1188 raft_consensus.cc:738] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.932778 1188 consensus_queue.cc:260] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.936340 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
W20250110 02:51:21.936486 2002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 02:51:21.936618 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 02:51:21.936681 2002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 02:51:21.936748 2002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 02:51:21.936640 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:21.937042 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:21.937633 1849 raft_consensus.cc:491] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:21.937765 1849 raft_consensus.cc:513] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.938118 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:21.938289 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:21.938442 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:21.938531 1849 leader_election.cc:290] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.939185 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:21.939359 775 raft_consensus.cc:2463] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.939908 996 leader_election.cc:304] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.940327 1849 raft_consensus.cc:2798] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:21.940410 1849 raft_consensus.cc:491] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:21.940470 1849 raft_consensus.cc:3054] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.940603 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:21.940768 914 raft_consensus.cc:2463] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:21.941495 1188 ts_tablet_manager.cc:1428] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:21.942523 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:21.942668 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:21.942902 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:21.943077 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:21.943203 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:21.943135 1849 raft_consensus.cc:513] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.943441 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:21.943810 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:21.944106 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:21.944168 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:21.944331 914 raft_consensus.cc:3054] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.944720 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871180863899419595cd632e7a4cf0dc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:21.944772 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:21.944916 775 raft_consensus.cc:3054] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:21.945050 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:21.945353 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:21.945574 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:21.945945 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:21.946329 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:21.946866 914 raft_consensus.cc:2463] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.947686 775 raft_consensus.cc:2463] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:21.948118 996 leader_election.cc:304] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:21.948428 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:21.948732 1680 raft_consensus.cc:2798] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:21.948889 1680 raft_consensus.cc:695] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:21.949193 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:21.949188 1680 consensus_queue.cc:237] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.949607 1457 catalog_manager.cc:5526] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:21.950030 1188 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.951555 1188 tablet_bootstrap.cc:654] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.951990 1849 leader_election.cc:290] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:21.951993 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:21.952713 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:21.953099 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:21.954561 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:21.955468 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 1 got 0
I20250110 02:51:21.958088 1184 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:21.958253 1184 ts_tablet_manager.cc:1397] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.003s
I20250110 02:51:21.958412 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:21.958701 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:21.959396 1184 raft_consensus.cc:357] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.959605 1184 raft_consensus.cc:383] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.959659 1184 raft_consensus.cc:738] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.959690 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:21.959931 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:21.959911 1184 consensus_queue.cc:260] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:21.960373 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:21.960886 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:21.961522 1184 ts_tablet_manager.cc:1428] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 02:51:21.961793 1184 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:21.961542 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:21.965080 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:21.965760 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:21.966063 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:21.966573 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:21.967023 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:21.967450 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:21.968178 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:21.969218 1188 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:21.969337 1188 ts_tablet_manager.cc:1397] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250110 02:51:21.970039 1188 raft_consensus.cc:357] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.970221 1188 raft_consensus.cc:383] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.970269 1188 raft_consensus.cc:738] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.970506 1188 consensus_queue.cc:260] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.974359 1188 ts_tablet_manager.cc:1428] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:21.974341 2024 consensus_queue.cc:1035] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:21.974638 1188 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:21.975513 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:21.976003 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:21.976725 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:21.978186 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:21.982791 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:21.984021 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:21.984923 1188 tablet_bootstrap.cc:654] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:21.985286 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
W20250110 02:51:21.986869 996 consensus_peers.cc:487] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c3ce95413df40369ee599176f91b892. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:21.988564 1181 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:21.988691 1181 ts_tablet_manager.cc:1397] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.002s
I20250110 02:51:21.989562 1181 raft_consensus.cc:357] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.989755 1181 raft_consensus.cc:383] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:21.989818 1181 raft_consensus.cc:738] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:21.990010 1181 consensus_queue.cc:260] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:21.990614 1181 ts_tablet_manager.cc:1428] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250110 02:51:21.990887 1181 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:21.993023 1181 tablet_bootstrap.cc:654] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.009301 1184 tablet_bootstrap.cc:654] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.010937 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:22.011358 2038 consensus_queue.cc:1035] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.011566 1181 tablet_bootstrap.cc:492] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.011696 1181 ts_tablet_manager.cc:1397] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.006s
I20250110 02:51:22.012329 1181 raft_consensus.cc:357] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.012490 1181 raft_consensus.cc:383] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.012548 1181 raft_consensus.cc:738] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.012780 1181 consensus_queue.cc:260] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.014457 1181 ts_tablet_manager.cc:1428] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:22.015252 1181 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.016670 1181 tablet_bootstrap.cc:654] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.019109 2002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35975,127.0.104.61:44449,127.0.104.60:39025
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=46707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 02:43:20 UTC on 24a791456cd2
build id 4103
ASAN enabled
I20250110 02:51:22.021276 2002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 02:51:22.023730 1188 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.023888 1188 ts_tablet_manager.cc:1397] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.013s
I20250110 02:51:22.026212 1188 raft_consensus.cc:357] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.026448 1188 raft_consensus.cc:383] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.026511 1188 raft_consensus.cc:738] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.026734 1188 consensus_queue.cc:260] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.029700 1188 ts_tablet_manager.cc:1428] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250110 02:51:22.030298 2002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250110 02:51:22.031307 1188 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.032929 1188 tablet_bootstrap.cc:654] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.034611 1181 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.034719 1181 ts_tablet_manager.cc:1397] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20250110 02:51:22.035421 1181 raft_consensus.cc:357] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.035601 1181 raft_consensus.cc:383] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.035645 1181 raft_consensus.cc:738] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.035847 1181 consensus_queue.cc:260] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.037200 1181 ts_tablet_manager.cc:1428] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:22.038195 1181 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.039786 1181 tablet_bootstrap.cc:654] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.043462 1184 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.043608 1184 ts_tablet_manager.cc:1397] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.082s user 0.019s sys 0.006s
I20250110 02:51:22.044483 1184 raft_consensus.cc:357] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.044726 1184 raft_consensus.cc:383] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.044793 1184 raft_consensus.cc:738] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.045035 1184 consensus_queue.cc:260] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.046972 1184 ts_tablet_manager.cc:1428] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:22.048151 1184 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.065730 1188 tablet_bootstrap.cc:492] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.065917 1188 ts_tablet_manager.cc:1397] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
I20250110 02:51:22.066900 1188 raft_consensus.cc:357] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.067078 1188 raft_consensus.cc:383] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.067152 1188 raft_consensus.cc:738] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.067396 1188 consensus_queue.cc:260] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.068055 1188 ts_tablet_manager.cc:1428] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250110 02:51:22.068480 1849 consensus_queue.cc:1035] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.082242 1188 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.085340 1990 consensus_queue.cc:1035] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.087204 2038 raft_consensus.cc:491] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.087323 2038 raft_consensus.cc:513] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.087865 2038 leader_election.cc:290] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:22.089725 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
W20250110 02:51:22.090227 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:22.090317 853 leader_election.cc:304] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.090662 2092 raft_consensus.cc:491] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.090785 2092 raft_consensus.cc:513] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.091310 2092 leader_election.cc:290] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.091720 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.091766 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:22.092195 711 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:22.092330 709 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.092449 709 leader_election.cc:304] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.092463 2038 raft_consensus.cc:2743] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.092921 2092 raft_consensus.cc:2743] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.095765 1188 tablet_bootstrap.cc:654] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.109472 2038 raft_consensus.cc:491] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.109640 2038 raft_consensus.cc:513] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.110435 2038 leader_election.cc:290] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.111210 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.111711 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa6057de9da04f2c82b8f25c5b4e2fe4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.112459 853 leader_election.cc:304] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.113020 2038 raft_consensus.cc:2743] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.114233 1188 tablet_bootstrap.cc:492] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.114400 1188 ts_tablet_manager.cc:1397] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.007s
I20250110 02:51:22.115458 1188 raft_consensus.cc:357] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.115674 1188 raft_consensus.cc:383] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.115718 1188 raft_consensus.cc:738] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.115996 1188 consensus_queue.cc:260] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.117097 1188 ts_tablet_manager.cc:1428] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:22.118417 1188 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.130559 2092 raft_consensus.cc:491] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.130759 1181 tablet_bootstrap.cc:492] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.130694 2092 raft_consensus.cc:513] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.130898 1181 ts_tablet_manager.cc:1397] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.093s user 0.023s sys 0.020s
I20250110 02:51:22.131243 2092 leader_election.cc:290] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.131913 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.132091 912 raft_consensus.cc:2463] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:22.132429 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.132654 1055 raft_consensus.cc:2463] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:22.133013 711 leader_election.cc:304] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.133299 2092 raft_consensus.cc:2798] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.133361 2092 raft_consensus.cc:491] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.133409 2092 raft_consensus.cc:3054] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.131768 1181 raft_consensus.cc:357] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.133699 1181 raft_consensus.cc:383] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.133769 1181 raft_consensus.cc:738] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.134016 1181 consensus_queue.cc:260] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.135733 1184 tablet_bootstrap.cc:654] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.135716 2092 raft_consensus.cc:513] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.136927 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.137073 912 raft_consensus.cc:3054] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.139798 912 raft_consensus.cc:2463] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:22.140377 709 leader_election.cc:304] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.141255 1849 consensus_queue.cc:1035] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.143187 2038 raft_consensus.cc:491] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.143302 2038 raft_consensus.cc:513] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.143841 2038 leader_election.cc:290] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.144444 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.144663 774 raft_consensus.cc:2388] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:22.145316 2092 leader_election.cc:290] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.145484 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fd370138ac4039be55642f1b4c5607" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:22.145648 1055 raft_consensus.cc:3054] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.148290 1055 raft_consensus.cc:2463] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:22.148562 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc9a4a842914a6f878d01194cd9a5f7" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.148782 2092 raft_consensus.cc:2798] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.148880 2092 raft_consensus.cc:695] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:22.148988 2038 raft_consensus.cc:491] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.149070 853 leader_election.cc:304] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.149101 2038 raft_consensus.cc:513] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.149380 2138 raft_consensus.cc:3054] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.149641 2038 leader_election.cc:290] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.150713 2092 consensus_queue.cc:237] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.152271 2138 raft_consensus.cc:2743] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.152788 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.152988 774 raft_consensus.cc:2463] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:22.153450 853 leader_election.cc:304] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.154075 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:22.154654 853 leader_election.cc:343] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.154943 2138 raft_consensus.cc:2798] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.155007 2138 raft_consensus.cc:491] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.155071 2138 raft_consensus.cc:3054] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.158242 1181 ts_tablet_manager.cc:1428] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250110 02:51:22.159184 2138 raft_consensus.cc:513] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.158541 1181 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.160972 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.161150 774 raft_consensus.cc:3054] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.161999 1181 tablet_bootstrap.cc:654] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.169353 2038 raft_consensus.cc:491] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.169458 2038 raft_consensus.cc:513] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.169958 2038 leader_election.cc:290] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.170737 774 raft_consensus.cc:2463] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:22.170997 853 leader_election.cc:343] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:22.171206 853 leader_election.cc:304] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.171622 2038 raft_consensus.cc:2798] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.171726 2038 raft_consensus.cc:695] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
W20250110 02:51:22.172044 853 leader_election.cc:343] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:22.171960 2038 consensus_queue.cc:237] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.172123 853 leader_election.cc:304] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.172361 1957 raft_consensus.cc:2743] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.175359 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
W20250110 02:51:22.175979 853 leader_election.cc:343] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.176287 2138 leader_election.cc:290] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.179591 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:22.181428 1990 raft_consensus.cc:491] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.181560 1990 raft_consensus.cc:513] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.181408 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:22.182296 1990 leader_election.cc:290] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.183483 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.183593 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
W20250110 02:51:22.184415 994 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.184697 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.184952 774 raft_consensus.cc:2463] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.185324 996 leader_election.cc:304] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.185734 1990 raft_consensus.cc:2798] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.185820 1990 raft_consensus.cc:491] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.185887 1990 raft_consensus.cc:3054] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.186022 2024 consensus_queue.cc:1035] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.185945 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:22.187400 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:22.188767 1990 raft_consensus.cc:513] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.189003 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:22.189998 1990 leader_election.cc:290] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.190275 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.190538 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.190726 772 raft_consensus.cc:3054] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.190958 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:22.193285 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:22.193873 772 raft_consensus.cc:2463] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:22.194563 994 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.196318 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:22.198298 996 leader_election.cc:304] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.198726 2024 raft_consensus.cc:2798] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.198836 2024 raft_consensus.cc:695] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.198727 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:22.199045 2024 consensus_queue.cc:237] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.200022 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:22.200424 2139 consensus_queue.cc:1035] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.203397 2055 consensus_queue.cc:1035] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.203584 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:22.204704 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:22.205677 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:22.207137 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:22.208833 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:22.210242 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:22.214093 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:22.215497 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:22.216491 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
W20250110 02:51:22.204499 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 64156.6us
I20250110 02:51:22.216223 2193 raft_consensus.cc:491] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.218359 2193 raft_consensus.cc:513] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.221014 1188 tablet_bootstrap.cc:654] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.222307 1184 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.222536 1184 ts_tablet_manager.cc:1397] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.174s user 0.007s sys 0.008s
I20250110 02:51:22.223173 1184 raft_consensus.cc:357] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.223544 1184 raft_consensus.cc:383] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.223646 1184 raft_consensus.cc:738] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.223701 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:22.224107 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:22.223932 1184 consensus_queue.cc:260] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.225371 1184 ts_tablet_manager.cc:1428] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:22.225679 1184 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.224545 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:22.226686 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:22.226876 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:22.227149 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:22.228019 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:22.228456 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:22.228678 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:22.228904 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:22.229343 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:22.229754 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:22.229902 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:22.230273 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:22.230417 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:22.230650 2138 consensus_queue.cc:1035] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.230860 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:22.231137 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:22.231501 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:22.231670 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:22.231918 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:22.232138 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:22.232343 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:22.232475 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:22.233083 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:22.233278 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:22.233515 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:22.233686 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:22.233842 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_bfbd91df5def489aa8830a6d7345e498 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 1 got 0
I20250110 02:51:22.235445 1181 tablet_bootstrap.cc:492] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.235637 1181 ts_tablet_manager.cc:1397] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.000s
I20250110 02:51:22.236510 1181 raft_consensus.cc:357] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.236930 1181 raft_consensus.cc:383] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.237044 1181 raft_consensus.cc:738] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.238955 1181 consensus_queue.cc:260] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.241456 1181 ts_tablet_manager.cc:1428] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:22.228828 2183 raft_consensus.cc:491] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.241938 2183 raft_consensus.cc:513] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.242591 2183 leader_election.cc:290] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:22.242638 1439 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24cf81039d5e4a47b426ea5d2f21acc2 consensus state has no leader
I20250110 02:51:22.242810 1439 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 43) took 1754 ms. Trace:
I20250110 02:51:22.242903 1439 rpcz_store.cc:276] 0110 02:51:20.488331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.175727 (+687396us) service_pool.cc:225] Handling call
0110 02:51:21.391503 (+215776us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:21.391722 (+ 219us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:21.391983 (+ 261us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:21.392044 (+ 61us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:21.573835 (+181791us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:22.242779 (+668944us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.450640 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.450758 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.450768 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:21.450772 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:21.450894 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.450904 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.450908 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:21.450966 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:21.450969 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:21.451001 (+ 32us) write_op.cc:270] Start()
0110 02:51:21.451044 (+ 43us) write_op.cc:276] Timestamp: P: 1736477481450946 usec, L: 0
0110 02:51:21.451048 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.451243 (+ 195us) log.cc:844] Serialized 405 byte log entry
0110 02:51:21.548287 (+ 97044us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.572538 (+ 24251us) write_op.cc:301] APPLY: starting
0110 02:51:21.572554 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.572590 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.572613 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.572662 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.572691 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:21.572698 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.572777 (+ 79us) log.cc:844] Serialized 23 byte log entry
0110 02:51:21.573421 (+ 644us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.573437 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:21.573443 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":24191,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":1399,"num_ops":1,"prepare.queue_time_us":58137,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":1320,"raft.queue_time_us":45,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":6,"replication_time_us":97201,"thread_start_us":37377,"threads_started":1}]]}
I20250110 02:51:22.243716 1184 tablet_bootstrap.cc:654] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.243187 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.245527 1181 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:22.245491 711 leader_election.cc:343] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.246726 2193 leader_election.cc:290] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.247233 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d970e2fe72b4727968ca605b3e772cc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.249025 1181 tablet_bootstrap.cc:654] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:22.247702 711 leader_election.cc:343] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.259334 1184 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.259482 1184 ts_tablet_manager.cc:1397] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20250110 02:51:22.260278 1184 raft_consensus.cc:357] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.260501 1184 raft_consensus.cc:383] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.260547 1184 raft_consensus.cc:738] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.260761 1184 consensus_queue.cc:260] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.261346 1184 ts_tablet_manager.cc:1428] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:22.261629 1184 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.270103 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.270323 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d970e2fe72b4727968ca605b3e772cc" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.268460 2139 consensus_queue.cc:1035] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:22.272796 709 leader_election.cc:343] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.272967 709 leader_election.cc:304] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:22.273360 709 leader_election.cc:343] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.273514 709 leader_election.cc:304] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.273864 2193 raft_consensus.cc:2743] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.274801 2193 raft_consensus.cc:2743] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.275827 912 raft_consensus.cc:3054] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.279810 1188 tablet_bootstrap.cc:492] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.279948 1188 ts_tablet_manager.cc:1397] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.025s
I20250110 02:51:22.297206 1181 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.297340 1181 ts_tablet_manager.cc:1397] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.001s
I20250110 02:51:22.298032 1181 raft_consensus.cc:357] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.298238 1181 raft_consensus.cc:383] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.298288 1181 raft_consensus.cc:738] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.298489 1181 consensus_queue.cc:260] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.298574 1990 raft_consensus.cc:491] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.298719 1990 raft_consensus.cc:513] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.299304 1990 leader_election.cc:290] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:22.300501 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:22.302938 1181 ts_tablet_manager.cc:1428] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:22.303232 1181 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.303398 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.303606 774 raft_consensus.cc:2388] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8899dd1aeb384e4ea4523f5fd225c888 in current term 1: Already voted for candidate 85f9597077934f1a9814d284ed06d13d in this term.
I20250110 02:51:22.304680 1184 tablet_bootstrap.cc:654] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.312156 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 09b174f3a2e74e51a4f815c220d21e8d (DEFAULT_TABLE table=default.Table_67bc8c66612446d8a0702142479b305e [id=ab8a3f8de6544300b1c286f27728680c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.312467 1990 consensus_queue.cc:1035] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.313099 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 66252189729442a38155efbacc0d25d7 (DEFAULT_TABLE table=default.Table_eeee9b94a4844bbdb5ced2be8c6cdaf3 [id=3a56acebf2ab4c52a7685890e51367a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.313728 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b174f3a2e74e51a4f815c220d21e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.313690 743 tablet_service.cc:1467] Processing CreateTablet for tablet 09b174f3a2e74e51a4f815c220d21e8d (DEFAULT_TABLE table=default.Table_67bc8c66612446d8a0702142479b305e [id=ab8a3f8de6544300b1c286f27728680c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.313987 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b174f3a2e74e51a4f815c220d21e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.314987 1181 tablet_bootstrap.cc:654] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.318367 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 439301830dff4b389451d709481bf1f0 (DEFAULT_TABLE table=default.Table_1fbfd53f85064ec8b3dd74e0a4d033c9 [id=f9e3028364904961bffe485c7886d3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.318791 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66252189729442a38155efbacc0d25d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.318807 747 tablet_service.cc:1467] Processing CreateTablet for tablet 66252189729442a38155efbacc0d25d7 (DEFAULT_TABLE table=default.Table_eeee9b94a4844bbdb5ced2be8c6cdaf3 [id=3a56acebf2ab4c52a7685890e51367a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.318976 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439301830dff4b389451d709481bf1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.319085 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66252189729442a38155efbacc0d25d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.322130 1041 tablet_service.cc:1467] Processing CreateTablet for tablet dba5a9b7e0fd4a28b18a8dbd451232bd (DEFAULT_TABLE table=default.Table_b031e9abfbe345358123f63c277edb99 [id=312d645594e64d82a5ad56bd0bfd53ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.322413 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba5a9b7e0fd4a28b18a8dbd451232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.323097 752 tablet_service.cc:1467] Processing CreateTablet for tablet 439301830dff4b389451d709481bf1f0 (DEFAULT_TABLE table=default.Table_1fbfd53f85064ec8b3dd74e0a4d033c9 [id=f9e3028364904961bffe485c7886d3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.323357 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439301830dff4b389451d709481bf1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.328775 751 tablet_service.cc:1467] Processing CreateTablet for tablet dba5a9b7e0fd4a28b18a8dbd451232bd (DEFAULT_TABLE table=default.Table_b031e9abfbe345358123f63c277edb99 [id=312d645594e64d82a5ad56bd0bfd53ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.329192 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba5a9b7e0fd4a28b18a8dbd451232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.335215 2199 raft_consensus.cc:491] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.335330 2199 raft_consensus.cc:513] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.335120 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 8303ee1c96cd485e8a191ac46a99b2cc (DEFAULT_TABLE table=default.Table_4f4e7744b0f5403891cb46da79efba4f [id=d2118d019a214fd489e5962b16335c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.335914 2199 leader_election.cc:290] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.336095 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8303ee1c96cd485e8a191ac46a99b2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.336045 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8303ee1c96cd485e8a191ac46a99b2cc (DEFAULT_TABLE table=default.Table_4f4e7744b0f5403891cb46da79efba4f [id=d2118d019a214fd489e5962b16335c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.336697 894 tablet_service.cc:1467] Processing CreateTablet for tablet 66252189729442a38155efbacc0d25d7 (DEFAULT_TABLE table=default.Table_eeee9b94a4844bbdb5ced2be8c6cdaf3 [id=3a56acebf2ab4c52a7685890e51367a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.336771 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598bc151d0c14fc6bb28523c535d9366" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.336413 1041 tablet_service.cc:1467] Processing CreateTablet for tablet 9b937188bf2949af8f54bade4e0ad984 (DEFAULT_TABLE table=default.Table_9ea14d3254a34f3cb6825fc2e407b2d7 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.336915 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8303ee1c96cd485e8a191ac46a99b2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.337131 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b937188bf2949af8f54bade4e0ad984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.337374 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ce9f6a3bb4e4a8d80ca0b047500a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:22.337944 711 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.343334 895 tablet_service.cc:1467] Processing CreateTablet for tablet 09b174f3a2e74e51a4f815c220d21e8d (DEFAULT_TABLE table=default.Table_67bc8c66612446d8a0702142479b305e [id=ab8a3f8de6544300b1c286f27728680c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.343966 994 leader_election.cc:304] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.344095 1041 tablet_service.cc:1467] Processing CreateTablet for tablet b45b91d084a44c1e8cdf1553967878e2 (DEFAULT_TABLE table=default.Table_610281233be84be48b3fd966dbbcdec2 [id=0496645179554a879747f54f45d22bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.344555 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9b937188bf2949af8f54bade4e0ad984 (DEFAULT_TABLE table=default.Table_9ea14d3254a34f3cb6825fc2e407b2d7 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.344610 2024 raft_consensus.cc:3054] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.345013 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b937188bf2949af8f54bade4e0ad984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.345466 747 tablet_service.cc:1467] Processing CreateTablet for tablet b45b91d084a44c1e8cdf1553967878e2 (DEFAULT_TABLE table=default.Table_610281233be84be48b3fd966dbbcdec2 [id=0496645179554a879747f54f45d22bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.345624 880 tablet_service.cc:1467] Processing CreateTablet for tablet 439301830dff4b389451d709481bf1f0 (DEFAULT_TABLE table=default.Table_1fbfd53f85064ec8b3dd74e0a4d033c9 [id=f9e3028364904961bffe485c7886d3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.345731 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45b91d084a44c1e8cdf1553967878e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.347175 2024 raft_consensus.cc:2743] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.348120 885 tablet_service.cc:1467] Processing CreateTablet for tablet dba5a9b7e0fd4a28b18a8dbd451232bd (DEFAULT_TABLE table=default.Table_b031e9abfbe345358123f63c277edb99 [id=312d645594e64d82a5ad56bd0bfd53ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.348896 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9b937188bf2949af8f54bade4e0ad984 (DEFAULT_TABLE table=default.Table_9ea14d3254a34f3cb6825fc2e407b2d7 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.350489 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45b91d084a44c1e8cdf1553967878e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.354229 891 tablet_service.cc:1467] Processing CreateTablet for tablet b45b91d084a44c1e8cdf1553967878e2 (DEFAULT_TABLE table=default.Table_610281233be84be48b3fd966dbbcdec2 [id=0496645179554a879747f54f45d22bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.354471 2038 raft_consensus.cc:491] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.354596 2038 raft_consensus.cc:513] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.355296 1680 consensus_queue.cc:1035] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.356114 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.356278 1054 raft_consensus.cc:2463] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:22.356981 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66252189729442a38155efbacc0d25d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.356591 853 leader_election.cc:304] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:22.359035 2024 consensus_queue.cc:1035] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.359985 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d468766a4d2b41308830865514f54a97 (DEFAULT_TABLE table=default.Table_d3ddbdeb213d4d65a47a41e1912df72d [id=cd708989caad4943ae9cb057d41e1a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.360525 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d468766a4d2b41308830865514f54a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.366364 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d7940a1574af1804d5f5139ca7494 (DEFAULT_TABLE table=default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1 [id=4e2fb95ceb8c43fc920b1cd9e73557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.366703 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d7940a1574af1804d5f5139ca7494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.369865 1181 tablet_bootstrap.cc:492] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.369972 1181 ts_tablet_manager.cc:1397] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250110 02:51:22.370471 1030 tablet_service.cc:1467] Processing CreateTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.370782 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860d60b3194dc5a2e7a94d39d4b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.370932 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d7940a1574af1804d5f5139ca7494 (DEFAULT_TABLE table=default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1 [id=4e2fb95ceb8c43fc920b1cd9e73557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.370946 1181 raft_consensus.cc:357] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.371109 1181 raft_consensus.cc:383] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.371157 1181 raft_consensus.cc:738] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.371243 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d7940a1574af1804d5f5139ca7494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.371367 1181 consensus_queue.cc:260] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.372529 1181 ts_tablet_manager.cc:1428] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:22.373365 1181 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.373759 747 tablet_service.cc:1467] Processing CreateTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.374627 1181 tablet_bootstrap.cc:654] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.374686 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860d60b3194dc5a2e7a94d39d4b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.374787 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.374944 776 raft_consensus.cc:2463] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:22.377681 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.378024 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f015e99f53f48599ac2d7a29220bc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.378497 896 tablet_service.cc:1467] Processing CreateTablet for tablet d468766a4d2b41308830865514f54a97 (DEFAULT_TABLE table=default.Table_d3ddbdeb213d4d65a47a41e1912df72d [id=cd708989caad4943ae9cb057d41e1a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.379902 2038 leader_election.cc:290] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.358837 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ce9f6a3bb4e4a8d80ca0b047500a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.388180 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:22.389093 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8303ee1c96cd485e8a191ac46a99b2cc (DEFAULT_TABLE table=default.Table_4f4e7744b0f5403891cb46da79efba4f [id=d2118d019a214fd489e5962b16335c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.389396 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b174f3a2e74e51a4f815c220d21e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.389634 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439301830dff4b389451d709481bf1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.389751 2139 raft_consensus.cc:2798] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.389849 2139 raft_consensus.cc:491] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.389959 2139 raft_consensus.cc:3054] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.392468 2139 raft_consensus.cc:513] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.393114 2139 leader_election.cc:290] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.393265 1832 raft_consensus.cc:491] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.393391 1832 raft_consensus.cc:513] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.393952 1832 leader_election.cc:290] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:22.394805 2181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:22.396955 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:22.397150 1054 raft_consensus.cc:3054] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.399941 1054 raft_consensus.cc:2463] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:22.400169 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.400367 1059 raft_consensus.cc:2463] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:22.401691 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6d57f3e9a44d178300d3505d504854" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.401785 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:22.401825 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 02:51:22.401934 771 raft_consensus.cc:2463] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:22.401943 774 raft_consensus.cc:3054] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.407130 2002 server_base.cc:1034] running on GCE node
I20250110 02:51:22.408193 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 02:51:22.408998 774 raft_consensus.cc:2463] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:22.410501 2199 raft_consensus.cc:491] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.410619 2199 raft_consensus.cc:513] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.411201 2199 leader_election.cc:290] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.411267 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b937188bf2949af8f54bade4e0ad984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.411296 853 leader_election.cc:304] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:22.412581 1181 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.412709 1181 ts_tablet_manager.cc:1397] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250110 02:51:22.414716 709 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.414850 709 leader_election.cc:304] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.416121 2199 raft_consensus.cc:2743] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.416388 751 tablet_service.cc:1467] Processing CreateTablet for tablet d468766a4d2b41308830865514f54a97 (DEFAULT_TABLE table=default.Table_d3ddbdeb213d4d65a47a41e1912df72d [id=cd708989caad4943ae9cb057d41e1a7a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:22.416858 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24cf81039d5e4a47b426ea5d2f21acc2 consensus state has no leader
I20250110 02:51:22.413407 1181 raft_consensus.cc:357] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.417353 1181 raft_consensus.cc:383] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.417419 1181 raft_consensus.cc:738] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.418109 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d468766a4d2b41308830865514f54a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.418740 1181 consensus_queue.cc:260] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.419023 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:22.419543 711 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.418730 1188 raft_consensus.cc:357] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.421514 1832 raft_consensus.cc:491] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.422353 2139 raft_consensus.cc:2798] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.422519 853 leader_election.cc:304] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.423892 886 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1d7940a1574af1804d5f5139ca7494 (DEFAULT_TABLE table=default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1 [id=4e2fb95ceb8c43fc920b1cd9e73557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.424575 1181 ts_tablet_manager.cc:1428] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250110 02:51:22.424940 1188 raft_consensus.cc:383] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.424923 883 tablet_service.cc:1467] Processing CreateTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.425441 1188 raft_consensus.cc:738] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.425827 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 02:51:22.425943 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45b91d084a44c1e8cdf1553967878e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.427429 2002 hybrid_clock.cc:648] HybridClock initialized: now 1736477482427332 us; error 75 us; skew 500 ppm
I20250110 02:51:22.428527 1181 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.428985 2002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 02:51:22.432080 1181 tablet_bootstrap.cc:654] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.432442 1184 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.432572 1184 ts_tablet_manager.cc:1397] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.171s user 0.006s sys 0.005s
I20250110 02:51:22.433146 1184 raft_consensus.cc:357] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.433322 1184 raft_consensus.cc:383] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.433378 1184 raft_consensus.cc:738] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.426976 2038 raft_consensus.cc:2798] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.434979 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba5a9b7e0fd4a28b18a8dbd451232bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.435003 1832 raft_consensus.cc:513] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.435889 2139 raft_consensus.cc:695] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:22.435813 1188 consensus_queue.cc:260] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.437798 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8303ee1c96cd485e8a191ac46a99b2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.437637 895 tablet_service.cc:1467] Processing CreateTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.436167 2139 consensus_queue.cc:237] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.439307 1184 consensus_queue.cc:260] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.439795 1184 ts_tablet_manager.cc:1428] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:22.440297 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.441041 709 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.441138 709 leader_election.cc:304] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.440104 2024 raft_consensus.cc:491] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.441730 2199 raft_consensus.cc:2743] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.441792 2024 raft_consensus.cc:513] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.438520 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860d60b3194dc5a2e7a94d39d4b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.442337 2024 leader_election.cc:290] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.442834 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d7940a1574af1804d5f5139ca7494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.443389 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.443698 2038 raft_consensus.cc:491] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.443819 2038 raft_consensus.cc:3054] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.443553 776 raft_consensus.cc:2463] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.444317 996 leader_election.cc:304] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.444545 2024 raft_consensus.cc:2798] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.444609 2024 raft_consensus.cc:491] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.444669 2024 raft_consensus.cc:3054] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.447300 2024 raft_consensus.cc:513] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.438971 1832 leader_election.cc:290] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.448347 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.448540 776 raft_consensus.cc:3054] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.452502 776 raft_consensus.cc:2463] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.452699 2024 leader_election.cc:290] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.452844 996 leader_election.cc:304] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.453106 2024 raft_consensus.cc:2798] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.453125 1188 ts_tablet_manager.cc:1428] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.167s user 0.003s sys 0.000s
I20250110 02:51:22.453397 1188 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.454226 2024 raft_consensus.cc:695] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
W20250110 02:51:22.454283 853 consensus_peers.cc:487] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:22.456061 1184 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.457156 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d468766a4d2b41308830865514f54a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.457748 1184 tablet_bootstrap.cc:654] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.458879 2024 consensus_queue.cc:237] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.464170 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.469728 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:22.470350 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:22.470602 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:22.470911 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:22.472020 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:22.472220 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:22.472448 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:22.472630 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
W20250110 02:51:22.471441 853 leader_election.cc:343] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:22.472826 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:22.473114 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:22.473115 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5ddbc6b02742609bc2ca84c863a9d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.473268 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:22.473426 914 raft_consensus.cc:2463] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.473773 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:22.473961 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:22.474220 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:22.474735 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:22.474915 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:22.475155 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:22.475450 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:22.475775 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:22.476068 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:22.476486 2038 raft_consensus.cc:513] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.478322 1188 tablet_bootstrap.cc:654] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.479265 2038 leader_election.cc:290] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:22.482054 853 leader_election.cc:343] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:22.482205 853 leader_election.cc:304] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.483110 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.483299 776 raft_consensus.cc:3054] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.483311 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e88d1bbc304b4ebbc849a7d783052e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:22.483485 1059 raft_consensus.cc:3054] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.483995 1832 raft_consensus.cc:2743] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.485898 776 raft_consensus.cc:2463] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:22.485976 1059 raft_consensus.cc:2463] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:22.488677 853 leader_election.cc:304] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.488677 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f015e99f53f48599ac2d7a29220bc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.498274 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:22.498458 2038 raft_consensus.cc:2798] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.492220 1439 catalog_manager.cc:5526] T 397f3261b76a4c82a3c265e68662841f P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:22.499222 1439 catalog_manager.cc:5526] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:22.500495 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:22.500663 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f015e99f53f48599ac2d7a29220bc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:22.500841 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:22.501065 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:22.501365 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:22.501700 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:22.502238 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:22.502421 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:22.502633 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:22.502832 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:22.503302 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]). Expected version 2 got 0
I20250110 02:51:22.503871 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:22.504048 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:22.504213 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:22.504765 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:22.505249 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:22.505542 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:22.505837 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:22.506038 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:22.513578 2038 raft_consensus.cc:695] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:22.514230 2038 consensus_queue.cc:237] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.516386 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:22.516913 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:22.517180 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:22.517360 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:22.520359 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:22.520538 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:22.521200 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:22.522037 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:22.522238 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:22.522426 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:22.526587 1443 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_38807d5e51d64c94b251680c200dc98a [id=02e6300bb79544e99ddcee13edb258ee] alter complete (version 1)
I20250110 02:51:22.527012 1441 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 4aa78a9ef5eb443da0c8983a7190440b
I20250110 02:51:22.527540 1441 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 56) took 1203 ms. Trace:
I20250110 02:51:22.527667 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:22.528061 2024 raft_consensus.cc:491] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.527638 1441 rpcz_store.cc:276] 0110 02:51:21.324150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.864697 (+540547us) service_pool.cc:225] Handling call
0110 02:51:21.865135 (+ 438us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:21.865241 (+ 106us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:21.865291 (+ 50us) catalog_manager.cc:2762] Locking tablets
0110 02:51:22.019012 (+153721us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:22.024846 (+ 5834us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:22.024982 (+ 136us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:22.526975 (+501993us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:22.527468 (+ 493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.019845 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.019989 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.020005 (+ 16us) write_op.cc:435] Acquired schema lock
0110 02:51:22.020009 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:22.020155 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.020171 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.020177 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.020301 (+ 124us) tablet.cc:698] Row locks acquired
0110 02:51:22.020305 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:22.020341 (+ 36us) write_op.cc:270] Start()
0110 02:51:22.020375 (+ 34us) write_op.cc:276] Timestamp: P: 1736477482020283 usec, L: 0
0110 02:51:22.020380 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.020558 (+ 178us) log.cc:844] Serialized 725 byte log entry
0110 02:51:22.023762 (+ 3204us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.024252 (+ 490us) write_op.cc:301] APPLY: starting
0110 02:51:22.024273 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.024327 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.024358 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.024449 (+ 91us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.024489 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:22.024500 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:22.024576 (+ 76us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.024638 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.024661 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:22.024670 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.024743 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.024747 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":424,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":488,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":824,"raft.queue_time_us":172,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":270,"replication_time_us":3353}]]}
I20250110 02:51:22.528185 2024 raft_consensus.cc:513] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.528801 2024 leader_election.cc:290] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.529279 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.529445 776 raft_consensus.cc:2463] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.529945 996 leader_election.cc:304] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.530454 1181 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.530570 1181 ts_tablet_manager.cc:1397] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.102s user 0.010s sys 0.014s
I20250110 02:51:22.531256 1181 raft_consensus.cc:357] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.531419 1181 raft_consensus.cc:383] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.531461 1181 raft_consensus.cc:738] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.531872 2024 raft_consensus.cc:2798] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.531936 2024 raft_consensus.cc:491] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.531992 2024 raft_consensus.cc:3054] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.535064 1188 tablet_bootstrap.cc:492] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.535183 1188 ts_tablet_manager.cc:1397] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.002s
I20250110 02:51:22.535040 1181 consensus_queue.cc:260] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.535432 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:22.535282 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.535866 1181 ts_tablet_manager.cc:1428] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:22.536266 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.536491 916 raft_consensus.cc:2463] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.536695 1188 raft_consensus.cc:357] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.536846 1188 raft_consensus.cc:383] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.536885 1188 raft_consensus.cc:738] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.537027 1188 consensus_queue.cc:260] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.537420 1188 ts_tablet_manager.cc:1428] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:22.537328 2024 raft_consensus.cc:513] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.538018 1188 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.538573 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.538796 776 raft_consensus.cc:3054] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.542147 776 raft_consensus.cc:2463] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.542651 996 leader_election.cc:304] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.542932 1849 raft_consensus.cc:2798] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.543044 1849 raft_consensus.cc:695] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.543303 1849 consensus_queue.cc:237] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.543885 1184 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.543992 1184 ts_tablet_manager.cc:1397] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.013s
I20250110 02:51:22.544642 1184 raft_consensus.cc:357] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.544799 1184 raft_consensus.cc:383] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.544844 1184 raft_consensus.cc:738] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.545042 1184 consensus_queue.cc:260] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.545734 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae5a6572bd4e7e90e92d4b1cf8d820" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.545897 916 raft_consensus.cc:3054] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.548439 916 raft_consensus.cc:2463] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.549255 1188 tablet_bootstrap.cc:654] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.549624 2024 leader_election.cc:290] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.551631 1181 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.553086 1184 ts_tablet_manager.cc:1428] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:22.557998 1181 tablet_bootstrap.cc:654] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.562443 1184 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.564008 1184 tablet_bootstrap.cc:654] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.574366 1181 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.575634 1449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_742e66ce36c0463a978cb00de2f01135 [id=b1aa3fd4123b4c6db9dbf8c5d10892ff] alter complete (version 3)
I20250110 02:51:22.575975 1181 ts_tablet_manager.cc:1397] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20250110 02:51:22.579013 1181 raft_consensus.cc:357] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.582304 2199 raft_consensus.cc:491] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.582464 2199 raft_consensus.cc:513] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.583197 2199 leader_election.cc:290] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.584561 1181 raft_consensus.cc:383] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.584800 1181 raft_consensus.cc:738] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.586674 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.587143 709 leader_election.cc:343] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.590821 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet d424b4da66184bd19eccefad80621516
I20250110 02:51:22.591387 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 57) took 1247 ms. Trace:
I20250110 02:51:22.591506 1448 rpcz_store.cc:276] 0110 02:51:21.344339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.869029 (+524690us) service_pool.cc:225] Handling call
0110 02:51:21.869417 (+ 388us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:21.869541 (+ 124us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:21.869599 (+ 58us) catalog_manager.cc:2762] Locking tablets
0110 02:51:21.869710 (+ 111us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:21.880521 (+ 10811us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:21.880591 (+ 70us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:22.590753 (+710162us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:22.591350 (+ 597us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.871151 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.871245 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.871251 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:21.871253 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.871357 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.871368 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.871372 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.871452 (+ 80us) tablet.cc:698] Row locks acquired
0110 02:51:21.871454 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.871480 (+ 26us) write_op.cc:270] Start()
0110 02:51:21.871504 (+ 24us) write_op.cc:276] Timestamp: P: 1736477481871441 usec, L: 0
0110 02:51:21.871507 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.871658 (+ 151us) log.cc:844] Serialized 725 byte log entry
0110 02:51:21.879316 (+ 7658us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.879626 (+ 310us) write_op.cc:301] APPLY: starting
0110 02:51:21.879643 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.879697 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.879721 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.879793 (+ 72us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.879822 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:21.879829 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.879890 (+ 61us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.879938 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.879956 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:21.879961 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:21.880027 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.880030 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":2,"prepare.queue_time_us":833,"prepare.run_cpu_time_us":589,"prepare.run_wall_time_us":589,"raft.queue_time_us":157,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":7782}]]}
I20250110 02:51:22.592218 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_18eb7bac666944ddaf84da77edd4462b" } modify_external_catalogs: true
I20250110 02:51:22.592417 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_18eb7bac666944ddaf84da77edd4462b" } modify_external_catalogs: true
I20250110 02:51:22.593012 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 35ec1dba54a247d5ba9f9e053aa3b645
I20250110 02:51:22.593488 1456 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 58) took 1241 ms. Trace:
I20250110 02:51:22.593573 1456 rpcz_store.cc:276] 0110 02:51:21.351734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.876098 (+524364us) service_pool.cc:225] Handling call
0110 02:51:21.876506 (+ 408us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:21.876624 (+ 118us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:21.876706 (+ 82us) catalog_manager.cc:2762] Locking tablets
0110 02:51:22.001020 (+124314us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:22.010802 (+ 9782us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:22.010882 (+ 80us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:22.592968 (+582086us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:22.593433 (+ 465us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.001936 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.002107 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.002117 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:22.002122 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:22.002261 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.002277 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.002284 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.002439 (+ 155us) tablet.cc:698] Row locks acquired
0110 02:51:22.002444 (+ 5us) write_op.cc:260] PREPARE: finished
0110 02:51:22.002482 (+ 38us) write_op.cc:270] Start()
0110 02:51:22.002516 (+ 34us) write_op.cc:276] Timestamp: P: 1736477482002426 usec, L: 0
0110 02:51:22.002521 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.002671 (+ 150us) log.cc:844] Serialized 759 byte log entry
0110 02:51:22.009075 (+ 6404us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.009738 (+ 663us) write_op.cc:301] APPLY: starting
0110 02:51:22.009755 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.009811 (+ 56us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.009851 (+ 40us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.009940 (+ 89us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.009986 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:22.010006 (+ 20us) write_op.cc:312] APPLY: finished
0110 02:51:22.010091 (+ 85us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.010144 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.010163 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:22.010169 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.010235 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.010238 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":590,"apply.run_cpu_time_us":499,"apply.run_wall_time_us":499,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":834,"prepare.run_wall_time_us":834,"raft.queue_time_us":155,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":250,"replication_time_us":6526}]]}
I20250110 02:51:22.594308 1456 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_f45dec555ce747f9be01f55fa8d8383f" } modify_external_catalogs: true
I20250110 02:51:22.594493 1456 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_f45dec555ce747f9be01f55fa8d8383f" } modify_external_catalogs: true
W20250110 02:51:22.595093 1450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6555b0a4491046bf8f5cd4ee406c3955 consensus state has no leader
I20250110 02:51:22.599323 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_26b5c54688724d26b6bf4f3587158e9c [id=9e880e06a6d648b7bc44b651084a14a3] alter complete (version 4)
I20250110 02:51:22.600960 895 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:22.601171 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:22.601467 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
W20250110 02:51:22.602329 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:22.585261 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.597433 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.597561 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.603246 711 leader_election.cc:343] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.603354 711 leader_election.cc:304] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.604038 2199 raft_consensus.cc:2743] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:22.604955 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d424b4da66184bd19eccefad80621516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d424b4da66184bd19eccefad80621516 already in progress: creating tablet
I20250110 02:51:22.605259 2224 tablet_replica.cc:331] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:22.605338 1188 tablet_bootstrap.cc:492] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.605463 1188 ts_tablet_manager.cc:1397] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.008s
I20250110 02:51:22.605571 2224 raft_consensus.cc:2238] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:22.607255 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_ed90cac8f4544e85bab3425999a77496 [id=621d3211bd1649a0aab67ae4c000f40d] alter complete (version 5)
I20250110 02:51:22.612571 1188 raft_consensus.cc:357] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.612816 1188 raft_consensus.cc:383] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.612915 1188 raft_consensus.cc:738] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.613147 1188 consensus_queue.cc:260] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.613871 1188 ts_tablet_manager.cc:1428] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250110 02:51:22.614184 1188 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.616741 1832 consensus_queue.cc:1035] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:22.619563 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
W20250110 02:51:22.621675 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d424b4da66184bd19eccefad80621516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d424b4da66184bd19eccefad80621516 already in progress: creating tablet
I20250110 02:51:22.623207 2222 tablet_replica.cc:331] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:22.623435 2222 raft_consensus.cc:2238] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:22.623625 2222 raft_consensus.cc:2267] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.624439 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_8115dcee1787457aab5b9023e08afe00 [id=0692620395c14593803592e6aade6af3] alter complete (version 1)
I20250110 02:51:22.627430 2222 ts_tablet_manager.cc:1905] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.627794 2224 raft_consensus.cc:2267] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.628813 2055 raft_consensus.cc:491] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.624344 1181 consensus_queue.cc:260] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.628940 2055 raft_consensus.cc:513] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.629351 1181 ts_tablet_manager.cc:1428] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20250110 02:51:22.629482 2055 leader_election.cc:290] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.630172 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.630518 776 raft_consensus.cc:2463] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.630978 996 leader_election.cc:304] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.631259 2024 raft_consensus.cc:2798] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.631363 2024 raft_consensus.cc:491] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.631460 2024 raft_consensus.cc:3054] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.632761 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:22.633145 2223 tablet_replica.cc:331] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:22.633460 2223 raft_consensus.cc:2238] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:22.633736 2223 raft_consensus.cc:2267] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.634189 2024 raft_consensus.cc:513] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.635313 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.635653 776 raft_consensus.cc:3054] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.638938 776 raft_consensus.cc:2463] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.639390 996 leader_election.cc:304] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.639642 2055 raft_consensus.cc:2798] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.639750 2055 raft_consensus.cc:695] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.639971 2055 consensus_queue.cc:237] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.644461 2024 leader_election.cc:290] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.630256 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.640588 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:22.646059 994 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:22.646540 994 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.648711 1188 tablet_bootstrap.cc:654] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.649816 2224 ts_tablet_manager.cc:1905] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.649976 1181 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.658252 2055 raft_consensus.cc:491] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.658331 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:22.658489 2055 raft_consensus.cc:513] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.659575 2055 leader_election.cc:290] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.660292 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.660696 994 leader_election.cc:343] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.660907 2055 consensus_queue.cc:1035] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.661055 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.661266 776 raft_consensus.cc:2463] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.661912 996 leader_election.cc:304] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.662506 2055 raft_consensus.cc:2798] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.662627 2055 raft_consensus.cc:491] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.662714 2055 raft_consensus.cc:3054] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.664075 2002 webserver.cc:458] Webserver started at http://127.0.104.61:46707/ using document root <none> and password file <none>
I20250110 02:51:22.664125 2199 raft_consensus.cc:491] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.664235 2199 raft_consensus.cc:513] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.664793 2199 leader_election.cc:290] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.665051 2002 fs_manager.cc:362] Metadata directory not provided
I20250110 02:51:22.665211 2002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 02:51:22.665210 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302af84fd8374ffa96a7503b54849536" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:22.665838 711 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.666231 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302af84fd8374ffa96a7503b54849536" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.666749 709 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.666849 709 leader_election.cc:304] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.667227 2199 raft_consensus.cc:2743] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.672976 1184 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:22.673193 1184 ts_tablet_manager.cc:1397] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.000s
I20250110 02:51:22.674407 1184 raft_consensus.cc:357] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.674669 1184 raft_consensus.cc:383] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.678352 2055 raft_consensus.cc:513] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.679313 2199 raft_consensus.cc:491] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.679426 2199 raft_consensus.cc:513] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.679961 2199 leader_election.cc:290] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.680315 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e05a7554a14805bdcbd1824a5bd9d5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.680493 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e05a7554a14805bdcbd1824a5bd9d5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.680786 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.680943 774 raft_consensus.cc:3054] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:22.681108 709 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.681520 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:22.681900 994 leader_election.cc:343] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.683214 774 raft_consensus.cc:2463] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:22.683538 711 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.683647 711 leader_election.cc:304] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.684111 2199 raft_consensus.cc:2743] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.684105 996 leader_election.cc:304] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.684298 1680 raft_consensus.cc:2798] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.684409 1680 raft_consensus.cc:695] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.684607 1181 tablet_bootstrap.cc:654] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.684537 1990 consensus_queue.cc:1035] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.684618 1680 consensus_queue.cc:237] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.692814 1184 raft_consensus.cc:738] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.694250 2224 ts_tablet_manager.cc:1918] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:22.694766 1184 consensus_queue.cc:260] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.697731 1181 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.697896 1181 ts_tablet_manager.cc:1397] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20250110 02:51:22.698812 1181 raft_consensus.cc:357] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.699040 1181 raft_consensus.cc:383] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.699097 1181 raft_consensus.cc:738] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.701385 2222 ts_tablet_manager.cc:1918] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:22.701323 1181 consensus_queue.cc:260] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.702957 1181 ts_tablet_manager.cc:1428] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250110 02:51:22.703231 1181 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.703996 2024 consensus_queue.cc:1035] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.707587 2224 log.cc:1198] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/35ec1dba54a247d5ba9f9e053aa3b645
I20250110 02:51:22.707949 2224 ts_tablet_manager.cc:1939] T 35ec1dba54a247d5ba9f9e053aa3b645 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:22.709023 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 35ec1dba54a247d5ba9f9e053aa3b645 (table default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]) successfully deleted
I20250110 02:51:22.709933 1181 tablet_bootstrap.cc:654] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.711391 2222 log.cc:1198] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/35ec1dba54a247d5ba9f9e053aa3b645
I20250110 02:51:22.711722 2222 ts_tablet_manager.cc:1939] T 35ec1dba54a247d5ba9f9e053aa3b645 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:22.714754 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 35ec1dba54a247d5ba9f9e053aa3b645 (table default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]) successfully deleted
I20250110 02:51:22.718065 1181 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.718183 1181 ts_tablet_manager.cc:1397] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20250110 02:51:22.718878 1181 raft_consensus.cc:357] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.719059 1181 raft_consensus.cc:383] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.719101 1181 raft_consensus.cc:738] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.721560 1181 consensus_queue.cc:260] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.723712 1181 ts_tablet_manager.cc:1428] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:22.723987 1181 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.732303 1181 tablet_bootstrap.cc:654] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.732545 1450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 41) took 2244 ms. Trace:
I20250110 02:51:22.732641 1450 rpcz_store.cc:276] 0110 02:51:20.488171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.149722 (+661551us) service_pool.cc:225] Handling call
0110 02:51:21.399452 (+249730us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:21.399650 (+ 198us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:21.399828 (+ 178us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:21.399898 (+ 70us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:21.598215 (+198317us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:22.732484 (+1134269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.453735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.453819 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.453824 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:21.453826 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.453893 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.453901 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.453904 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:21.453944 (+ 40us) tablet.cc:698] Row locks acquired
0110 02:51:21.453946 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.453968 (+ 22us) write_op.cc:270] Start()
0110 02:51:21.453990 (+ 22us) write_op.cc:276] Timestamp: P: 1736477481453915 usec, L: 0
0110 02:51:21.453995 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.454140 (+ 145us) log.cc:844] Serialized 405 byte log entry
0110 02:51:21.585999 (+131859us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.597733 (+ 11734us) write_op.cc:301] APPLY: starting
0110 02:51:21.597743 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.597770 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.597802 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.597836 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.597860 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:21.597867 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.597916 (+ 49us) log.cc:844] Serialized 23 byte log entry
0110 02:51:21.598133 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.598146 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:21.598152 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11705,"apply.run_cpu_time_us":304,"apply.run_wall_time_us":558,"num_ops":1,"prepare.queue_time_us":53480,"prepare.run_cpu_time_us":481,"prepare.run_wall_time_us":798,"raft.queue_time_us":29,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":131991,"thread_start_us":19812,"threads_started":1}]]}
I20250110 02:51:22.733101 1184 ts_tablet_manager.cc:1428] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20250110 02:51:22.733331 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet 7564208da99449cd97338a41b263e750 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:22.715040 2199 raft_consensus.cc:491] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.717284 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:22.733592 2199 raft_consensus.cc:513] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.734177 2199 leader_election.cc:290] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.734817 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 7e554d00d05b4fa7a14c9b6159f9d726
I20250110 02:51:22.735392 1456 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 60) took 1358 ms. Trace:
I20250110 02:51:22.735510 1456 rpcz_store.cc:276] 0110 02:51:21.377001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.594152 (+1217151us) service_pool.cc:225] Handling call
0110 02:51:22.594593 (+ 441us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:22.594729 (+ 136us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:22.594798 (+ 69us) catalog_manager.cc:2762] Locking tablets
0110 02:51:22.652089 (+ 57291us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:22.657379 (+ 5290us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:22.734660 (+ 77281us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:22.734745 (+ 85us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:22.735351 (+ 606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.652799 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.652888 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.652895 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:22.652897 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.653006 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.653020 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.653025 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.653130 (+ 105us) tablet.cc:698] Row locks acquired
0110 02:51:22.653132 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.653163 (+ 31us) write_op.cc:270] Start()
0110 02:51:22.653190 (+ 27us) write_op.cc:276] Timestamp: P: 1736477482653141 usec, L: 0
0110 02:51:22.653193 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.653349 (+ 156us) log.cc:844] Serialized 756 byte log entry
0110 02:51:22.656624 (+ 3275us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.656767 (+ 143us) write_op.cc:301] APPLY: starting
0110 02:51:22.656780 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.656818 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.656839 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.656920 (+ 81us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.656950 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:22.656957 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.657021 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.657073 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.657099 (+ 26us) write_op.cc:454] Released schema lock
0110 02:51:22.657104 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.657162 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.657166 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":389,"num_ops":2,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":957,"raft.queue_time_us":103,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":217,"replication_time_us":3407}]]}
I20250110 02:51:22.736431 1456 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_d9a1b5e10e2040c2b162e4fa3414d9fa" } new_table_name: "default.Table_5bef6387b90c461e8266e7b3052ee939" modify_external_catalogs: true
I20250110 02:51:22.737061 1181 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.737183 1181 ts_tablet_manager.cc:1397] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250110 02:51:22.737993 1181 raft_consensus.cc:357] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.738219 1181 raft_consensus.cc:383] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.738276 1181 raft_consensus.cc:738] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.738876 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:22.739009 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.739177 914 raft_consensus.cc:2463] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:22.739475 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7e554d00d05b4fa7a14c9b6159f9d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e554d00d05b4fa7a14c9b6159f9d726 already in progress: creating tablet
I20250110 02:51:22.739768 709 leader_election.cc:304] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.740051 2198 raft_consensus.cc:2798] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.740125 2198 raft_consensus.cc:491] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.740238 2198 raft_consensus.cc:3054] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.740608 1181 consensus_queue.cc:260] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.741128 1181 ts_tablet_manager.cc:1428] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:22.741400 1181 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.742954 2198 raft_consensus.cc:513] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.743549 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.743702 1059 raft_consensus.cc:2463] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:22.744508 2139 raft_consensus.cc:491] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.744649 2139 raft_consensus.cc:513] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.745254 2139 leader_election.cc:290] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.746651 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.746855 914 raft_consensus.cc:3054] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.746912 1181 tablet_bootstrap.cc:654] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.749481 914 raft_consensus.cc:2463] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
W20250110 02:51:22.750029 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:22.750485 709 leader_election.cc:304] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:22.751186 2199 raft_consensus.cc:2798] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.751780 2198 leader_election.cc:290] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.752694 2199 raft_consensus.cc:695] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:22.752933 2199 consensus_queue.cc:237] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.753422 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79175853474cbea951f5a5686bf630" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:22.753576 1059 raft_consensus.cc:3054] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.755424 1059 raft_consensus.cc:2463] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:22.755717 1054 raft_consensus.cc:3054] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.756423 914 raft_consensus.cc:3054] T 6e0588e5e5714cc5bd1c73665e5645b1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.756453 1181 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.756568 1181 ts_tablet_manager.cc:1397] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250110 02:51:22.757354 1181 raft_consensus.cc:357] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.757517 1181 raft_consensus.cc:383] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.757572 1181 raft_consensus.cc:738] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.758272 2139 consensus_queue.cc:1035] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.758992 1456 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 62) took 1163 ms. Trace:
I20250110 02:51:22.759078 1456 rpcz_store.cc:276] 0110 02:51:21.595403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.736231 (+1140828us) service_pool.cc:225] Handling call
0110 02:51:22.736623 (+ 392us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:22.736858 (+ 235us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:22.737044 (+ 186us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:22.737099 (+ 55us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:22.758488 (+ 21389us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:22.758936 (+ 448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.738123 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.738218 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.738224 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:22.738227 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:22.738323 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.738336 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.738340 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.738415 (+ 75us) tablet.cc:698] Row locks acquired
0110 02:51:22.738418 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:22.738451 (+ 33us) write_op.cc:270] Start()
0110 02:51:22.738481 (+ 30us) write_op.cc:276] Timestamp: P: 1736477482738420 usec, L: 0
0110 02:51:22.738484 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.738697 (+ 213us) log.cc:844] Serialized 405 byte log entry
0110 02:51:22.750735 (+ 12038us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.750853 (+ 118us) write_op.cc:301] APPLY: starting
0110 02:51:22.750869 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.750908 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.750938 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.750994 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.751028 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.751036 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:22.751119 (+ 83us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.751192 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.751212 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:22.751218 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.751293 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.751299 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":428,"apply.run_wall_time_us":428,"num_ops":1,"prepare.queue_time_us":465,"prepare.run_cpu_time_us":692,"prepare.run_wall_time_us":691,"raft.queue_time_us":8360,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":209,"replication_time_us":12224}]]}
I20250110 02:51:22.759855 1456 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e49b14cd5bac4073be1ad6ffe44eb92c" } new_table_name: "default.Table_d6c1d16cc59d472dabdbb7be7378be29" modify_external_catalogs: true
I20250110 02:51:22.760365 1849 consensus_queue.cc:1035] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:22.762987 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:22.763098 853 leader_election.cc:304] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.763283 1832 raft_consensus.cc:2743] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.763278 1181 consensus_queue.cc:260] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.765059 1181 ts_tablet_manager.cc:1428] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:22.765354 1181 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.770418 1849 consensus_queue.cc:1035] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.772747 1181 tablet_bootstrap.cc:654] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.774206 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.774951 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7e554d00d05b4fa7a14c9b6159f9d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e554d00d05b4fa7a14c9b6159f9d726 already in progress: creating tablet
I20250110 02:51:22.786824 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.787660 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
I20250110 02:51:22.787880 1181 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:22.788030 1181 ts_tablet_manager.cc:1397] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250110 02:51:22.789744 1181 raft_consensus.cc:357] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.789979 1181 raft_consensus.cc:383] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.790050 1181 raft_consensus.cc:738] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.732709 2223 ts_tablet_manager.cc:1905] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.791467 1181 consensus_queue.cc:260] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.792486 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.792719 2224 tablet_replica.cc:331] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:22.792688 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e03d53860ab744968693cf96467c3a19 consensus state has no leader
I20250110 02:51:22.792891 2224 raft_consensus.cc:2238] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:22.793335 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c1608d90ceb64602889b5705ba4ab55d consensus state has no leader
I20250110 02:51:22.792505 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.798743 2224 pending_rounds.cc:62] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:22.798818 2224 pending_rounds.cc:69] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611768158212096 op_type: NO_OP noop_request { }
I20250110 02:51:22.798923 2224 raft_consensus.cc:2883] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:22.799012 2224 raft_consensus.cc:2267] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.800289 2224 ts_tablet_manager.cc:1905] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.801687 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet b4f1782c46a44ed091f4c533226c71f2
I20250110 02:51:22.802222 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 59) took 1427 ms. Trace:
I20250110 02:51:22.802338 896 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:22.802901 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:22.802333 1448 rpcz_store.cc:276] 0110 02:51:21.374321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.592054 (+1217733us) service_pool.cc:225] Handling call
0110 02:51:22.592519 (+ 465us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:22.592652 (+ 133us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:22.592709 (+ 57us) catalog_manager.cc:2762] Locking tablets
0110 02:51:22.794292 (+201583us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:22.801510 (+ 7218us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:22.801594 (+ 84us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:22.801647 (+ 53us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:22.802163 (+ 516us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.795104 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.795208 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.795216 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:22.795219 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:22.795335 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.795352 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.795359 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.795494 (+ 135us) tablet.cc:698] Row locks acquired
0110 02:51:22.795498 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:22.795535 (+ 37us) write_op.cc:270] Start()
0110 02:51:22.795570 (+ 35us) write_op.cc:276] Timestamp: P: 1736477482795498 usec, L: 0
0110 02:51:22.795575 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.795761 (+ 186us) log.cc:844] Serialized 725 byte log entry
0110 02:51:22.800903 (+ 5142us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.801004 (+ 101us) write_op.cc:301] APPLY: starting
0110 02:51:22.801017 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.801060 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.801099 (+ 39us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.801160 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.801190 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:22.801197 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.801262 (+ 65us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.801315 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.801334 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:22.801339 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.801402 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.801405 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":388,"apply.run_wall_time_us":387,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":754,"raft.queue_time_us":1735,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":243,"replication_time_us":5302}]]}
I20250110 02:51:22.804132 1448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_ed90cac8f4544e85bab3425999a77496" } new_table_name: "default.Table_ed1ab72a595140a98cef7ad004d2c981" modify_external_catalogs: true
I20250110 02:51:22.806993 2224 ts_tablet_manager.cc:1918] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:22.809247 2224 log.cc:1198] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:22.810230 2224 ts_tablet_manager.cc:1939] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:22.813858 1181 ts_tablet_manager.cc:1428] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250110 02:51:22.814669 1181 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:22.814754 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (table default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]) successfully deleted
W20250110 02:51:22.814976 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:22.817422 1181 tablet_bootstrap.cc:654] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:22.826344 1448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 64) took 1221 ms. Trace:
I20250110 02:51:22.826490 1448 rpcz_store.cc:276] 0110 02:51:21.604845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.803928 (+1199083us) service_pool.cc:225] Handling call
0110 02:51:22.804333 (+ 405us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:22.804575 (+ 242us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:22.804779 (+ 204us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:22.804862 (+ 83us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:22.825899 (+ 21037us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:22.826303 (+ 404us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.805902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.806010 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.806018 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:22.806022 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:22.806148 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.806163 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.806170 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.806237 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:22.806242 (+ 5us) write_op.cc:260] PREPARE: finished
0110 02:51:22.806275 (+ 33us) write_op.cc:270] Start()
0110 02:51:22.806307 (+ 32us) write_op.cc:276] Timestamp: P: 1736477482806237 usec, L: 0
0110 02:51:22.806313 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.806499 (+ 186us) log.cc:844] Serialized 405 byte log entry
0110 02:51:22.824625 (+ 18126us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.824761 (+ 136us) write_op.cc:301] APPLY: starting
0110 02:51:22.824780 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.824825 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.824863 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.824921 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.824961 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.824971 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:22.825067 (+ 96us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.825131 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.825152 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:22.825160 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.825228 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.825234 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":460,"num_ops":1,"prepare.queue_time_us":555,"prepare.run_cpu_time_us":668,"prepare.run_wall_time_us":668,"replication_time_us":18273}]]}
I20250110 02:51:22.827304 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_b1de670e3fe347108cccbf69ab116d85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:22.827776 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:22.827890 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1de670e3fe347108cccbf69ab116d85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:22.837072 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.837831 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:22.842675 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 65) took 1045 ms. Trace:
I20250110 02:51:22.842876 1448 rpcz_store.cc:276] 0110 02:51:21.796897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.827086 (+1030189us) service_pool.cc:225] Handling call
0110 02:51:22.828001 (+ 915us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:22.828558 (+ 557us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:22.842429 (+ 13871us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:22.842541 (+ 112us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:22.842637 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.829579 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.829674 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.829679 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:22.829681 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.829772 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.829784 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.829789 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.829857 (+ 68us) tablet.cc:698] Row locks acquired
0110 02:51:22.829859 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.829891 (+ 32us) write_op.cc:270] Start()
0110 02:51:22.829920 (+ 29us) write_op.cc:276] Timestamp: P: 1736477482829851 usec, L: 0
0110 02:51:22.829926 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.830085 (+ 159us) log.cc:844] Serialized 433 byte log entry
0110 02:51:22.834867 (+ 4782us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.835005 (+ 138us) write_op.cc:301] APPLY: starting
0110 02:51:22.835022 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.835064 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.835091 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.835202 (+ 111us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.835235 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:22.835242 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.835314 (+ 72us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.835371 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.835388 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:22.835395 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.835493 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.835500 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":471,"apply.run_wall_time_us":470,"num_ops":2,"prepare.queue_time_us":489,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":572,"replication_time_us":4912}]]}
I20250110 02:51:22.843935 2038 consensus_queue.cc:1035] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:22.849397 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:22.852907 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_f3ea92dbdb6e4f3dbc723ffac6b602a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:22.853621 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3ea92dbdb6e4f3dbc723ffac6b602a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:22.856032 1184 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:22.866154 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.867096 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
I20250110 02:51:22.873875 2024 raft_consensus.cc:491] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.874038 2024 raft_consensus.cc:513] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.874660 2024 leader_election.cc:290] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.875295 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.875412 2258 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
W20250110 02:51:22.875720 994 leader_election.cc:343] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.883114 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.883530 2224 tablet_replica.cc:331] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:22.883720 2224 raft_consensus.cc:2238] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:22.884032 2224 raft_consensus.cc:2267] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.884402 1188 tablet_bootstrap.cc:492] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:22.884508 2224 ts_tablet_manager.cc:1905] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.884516 1188 ts_tablet_manager.cc:1397] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.270s user 0.025s sys 0.033s
I20250110 02:51:22.885381 1188 raft_consensus.cc:357] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.885568 1188 raft_consensus.cc:383] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:22.885617 1188 raft_consensus.cc:738] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:22.885901 1188 consensus_queue.cc:260] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.893637 2224 ts_tablet_manager.cc:1918] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:22.893896 2224 log.cc:1198] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/46a4745356e3423f9d4892d242da1c48
I20250110 02:51:22.894537 1849 raft_consensus.cc:491] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.894685 1849 raft_consensus.cc:513] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.895015 2224 ts_tablet_manager.cc:1939] T 46a4745356e3423f9d4892d242da1c48 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:22.895306 1849 leader_election.cc:290] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.896015 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:22.896567 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 46a4745356e3423f9d4892d242da1c48 (table default.Table_123de8ace5ed47f7850040c5de4fcbc7 [id=3aad422d584f40df9d9241586adbc66f]) successfully deleted
W20250110 02:51:22.896867 994 leader_election.cc:343] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.897182 2198 raft_consensus.cc:491] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.897320 2198 raft_consensus.cc:513] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.899760 2055 leader_election.cc:290] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.899820 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.904114 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.904296 767 raft_consensus.cc:2463] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.905114 2224 tablet_replica.cc:331] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:22.905398 2224 raft_consensus.cc:2238] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:22.905601 2224 raft_consensus.cc:2267] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:22.906131 2224 ts_tablet_manager.cc:1905] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:22.906277 2198 leader_election.cc:290] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.906765 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.906945 764 raft_consensus.cc:2463] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:22.907109 996 leader_election.cc:304] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.907728 2024 raft_consensus.cc:2798] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.907817 2024 raft_consensus.cc:491] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.907886 2024 raft_consensus.cc:3054] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.907964 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.908512 709 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.909075 996 leader_election.cc:304] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.909425 1849 raft_consensus.cc:2798] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.909490 1849 raft_consensus.cc:491] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.909551 1849 raft_consensus.cc:3054] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.910696 2024 raft_consensus.cc:513] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.911464 2024 leader_election.cc:290] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.911993 1849 raft_consensus.cc:513] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.912529 1849 leader_election.cc:290] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:22.912845 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:22.912994 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:22.913246 994 leader_election.cc:343] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.913489 1188 ts_tablet_manager.cc:1428] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
W20250110 02:51:22.913558 994 leader_election.cc:343] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.914516 1188 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:22.916254 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.916407 1188 tablet_bootstrap.cc:654] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:22.912365 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ce4fe51f2e914865a2e8bbaa256b8098 consensus state has no leader
W20250110 02:51:22.917488 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a4c6de957fd14af983fc1a3cd8629226 consensus state has no leader
W20250110 02:51:22.917706 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d3858638b6fc48d0bb8b8f97370a791d consensus state has no leader
W20250110 02:51:22.917886 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 75ca5369caf3415ea508352362409656 consensus state has no leader
W20250110 02:51:22.918047 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 00ebf8debb9d4394839d1c1cb97820a5 consensus state has no leader
I20250110 02:51:22.918757 2257 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250110 02:51:22.918829 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.929806 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 815767a4c7624852bfedab9fa6cb5f44 consensus state has no leader
I20250110 02:51:22.930516 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.930783 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.931169 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ae3033595f824affb2d300c3964e1e8e consensus state has no leader
I20250110 02:51:22.931748 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.933019 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.933300 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1edb4e06e40c43438882e1eb622d788e consensus state has no leader
I20250110 02:51:22.934357 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.935413 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:22.935837 2198 raft_consensus.cc:491] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:22.935885 711 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.936010 711 leader_election.cc:304] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:22.935976 2198 raft_consensus.cc:513] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.936445 2186 raft_consensus.cc:2743] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.937158 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:22.937507 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet abbd43638a2c47dcba4f6f90e8676f42 consensus state has no leader
I20250110 02:51:22.938875 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.939074 764 raft_consensus.cc:3054] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.939930 2198 leader_election.cc:290] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.940539 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.941027 709 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.941875 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:22.942023 767 raft_consensus.cc:3054] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.943264 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.943526 1054 raft_consensus.cc:2463] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:22.944205 711 leader_election.cc:304] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.944602 2186 raft_consensus.cc:2798] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:22.944931 2186 raft_consensus.cc:491] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:22.945030 2186 raft_consensus.cc:3054] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.951121 2138 raft_consensus.cc:491] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.951241 2138 raft_consensus.cc:513] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.951793 2138 leader_election.cc:290] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.952255 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:22.952452 765 raft_consensus.cc:2388] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:22.952527 767 raft_consensus.cc:2463] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.953083 996 leader_election.cc:304] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.953454 2240 raft_consensus.cc:2798] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.953588 2240 raft_consensus.cc:695] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.953972 2240 consensus_queue.cc:237] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:22.958971 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f1d916a5ea48708c2c1a61c92fe137" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:22.959512 853 leader_election.cc:304] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:22.961081 2138 raft_consensus.cc:3054] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.963765 2138 raft_consensus.cc:2743] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:22.972672 764 raft_consensus.cc:2463] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:22.973615 996 leader_election.cc:304] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.973976 2186 raft_consensus.cc:513] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.974442 2199 raft_consensus.cc:491] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:22.974596 2240 raft_consensus.cc:2798] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.974592 2199 raft_consensus.cc:513] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.974805 2240 raft_consensus.cc:695] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:22.975229 2240 consensus_queue.cc:237] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:22.979835 2002 fs_manager.cc:714] Time spent opening directory manager: real 0.312s user 0.030s sys 0.036s
I20250110 02:51:22.981865 2186 leader_election.cc:290] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:22.981793 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:22.982326 709 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.983512 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:22.983697 1054 raft_consensus.cc:3054] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:22.986388 1054 raft_consensus.cc:2463] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:22.986881 711 leader_election.cc:304] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:22.991621 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de9cf980d9c4117b4ec3ca937cfcf01" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:22.992169 709 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:22.992754 2253 raft_consensus.cc:2798] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:22.992981 2253 raft_consensus.cc:695] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:22.993923 2257 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:22.997962 2223 ts_tablet_manager.cc:1918] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:22.999751 2223 log.cc:1198] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/35ec1dba54a247d5ba9f9e053aa3b645
I20250110 02:51:23.002604 2199 leader_election.cc:290] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.002622 2240 raft_consensus.cc:491] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.002799 2240 raft_consensus.cc:513] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:23.003428 1434 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:44449, user_credentials={real_user=slave}} blocked reactor thread for 91379.1us
I20250110 02:51:23.003727 2240 leader_election.cc:290] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.003710 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.004207 994 leader_election.cc:343] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.007184 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.007350 764 raft_consensus.cc:2463] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.007606 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de9cf980d9c4117b4ec3ca937cfcf01" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.007648 996 leader_election.cc:304] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:23.009505 2240 raft_consensus.cc:2798] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.009644 2240 raft_consensus.cc:491] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250110 02:51:23.009689 711 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.009784 2240 raft_consensus.cc:3054] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.009900 711 leader_election.cc:304] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.013226 2240 raft_consensus.cc:513] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.017735 2199 raft_consensus.cc:2743] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.018929 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:23.019445 994 leader_election.cc:343] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.020855 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b375a932156d4abb908dbf327689b0f9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.021006 764 raft_consensus.cc:3054] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.022310 1188 tablet_bootstrap.cc:492] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.022542 1188 ts_tablet_manager.cc:1397] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.004s
I20250110 02:51:23.023404 1188 raft_consensus.cc:357] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.023536 764 raft_consensus.cc:2463] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.023872 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 7564208da99449cd97338a41b263e750
I20250110 02:51:23.023902 996 leader_election.cc:304] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:23.024173 2024 raft_consensus.cc:2798] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.024264 2024 raft_consensus.cc:695] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.024385 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 66) took 1225 ms. Trace:
I20250110 02:51:23.024467 2024 consensus_queue.cc:237] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.024787 1450 catalog_manager.cc:6596] ReplaceTablet: tablet 7564208da99449cd97338a41b263e750 deleted and replaced by tablet 6598311f59c6476ba339b8e57961cf67
I20250110 02:51:23.024904 1450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 61) took 1622 ms. Trace:
I20250110 02:51:23.024983 1450 rpcz_store.cc:276] 0110 02:51:21.402667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.733251 (+1330584us) service_pool.cc:225] Handling call
0110 02:51:23.024878 (+291627us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.870751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.870866 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.870872 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:22.870874 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.871015 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.871028 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.871033 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.871103 (+ 70us) tablet.cc:698] Row locks acquired
0110 02:51:22.871106 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:22.871137 (+ 31us) write_op.cc:270] Start()
0110 02:51:22.871166 (+ 29us) write_op.cc:276] Timestamp: P: 1736477482871094 usec, L: 0
0110 02:51:22.871170 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.871343 (+ 173us) log.cc:844] Serialized 622 byte log entry
0110 02:51:22.875238 (+ 3895us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.875345 (+ 107us) write_op.cc:301] APPLY: starting
0110 02:51:22.875360 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.875401 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.875424 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.875512 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.875581 (+ 69us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.875589 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:22.875669 (+ 80us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.875727 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.875744 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:22.875750 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.875825 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.875829 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":468,"apply.run_wall_time_us":466,"num_ops":2,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":689,"raft.queue_time_us":63,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":215,"replication_time_us":4043}]]}
I20250110 02:51:23.026280 1181 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.025897 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_c34337ea54d740158f4bfc9c10db9a5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:23.026405 1181 ts_tablet_manager.cc:1397] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.212s user 0.020s sys 0.032s
I20250110 02:51:23.024492 1448 rpcz_store.cc:276] 0110 02:51:21.799287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.852646 (+1053359us) service_pool.cc:225] Handling call
0110 02:51:22.854617 (+ 1971us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:22.901485 (+ 46868us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:22.958096 (+ 56611us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.024236 (+ 66140us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.024340 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.902470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.902602 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.902611 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:22.902615 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:22.902732 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.902895 (+ 163us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.902904 (+ 9us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.903003 (+ 99us) tablet.cc:698] Row locks acquired
0110 02:51:22.903006 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:22.903063 (+ 57us) write_op.cc:270] Start()
0110 02:51:22.903098 (+ 35us) write_op.cc:276] Timestamp: P: 1736477482903008 usec, L: 0
0110 02:51:22.903104 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.903329 (+ 225us) log.cc:844] Serialized 433 byte log entry
0110 02:51:22.950843 (+ 47514us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.950945 (+ 102us) write_op.cc:301] APPLY: starting
0110 02:51:22.950963 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.951018 (+ 55us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.951055 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.951160 (+ 105us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.951192 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:22.951199 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.951276 (+ 77us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.951334 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.951355 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:22.951362 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.951426 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.951430 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":471,"apply.run_wall_time_us":471,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":864,"prepare.run_wall_time_us":1921,"raft.queue_time_us":293,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":232,"replication_time_us":47708}]]}
I20250110 02:51:23.027047 1181 raft_consensus.cc:357] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.027220 1181 raft_consensus.cc:383] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.027267 1181 raft_consensus.cc:738] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:23.027348 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c34337ea54d740158f4bfc9c10db9a5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.027452 1181 consensus_queue.cc:260] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.027671 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_119ec4ba01fd4c019b13cd28b0a3b82c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:23.028153 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_119ec4ba01fd4c019b13cd28b0a3b82c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.028213 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:23.028645 1181 ts_tablet_manager.cc:1428] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:23.028820 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.029312 1181 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.030920 1181 tablet_bootstrap.cc:654] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.033876 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 67) took 1211 ms. Trace:
I20250110 02:51:23.034013 1450 rpcz_store.cc:276] 0110 02:51:21.822313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.025659 (+1203346us) service_pool.cc:225] Handling call
0110 02:51:23.027528 (+ 1869us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.028020 (+ 492us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.033591 (+ 5571us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.033739 (+ 148us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.033829 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.028811 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.028933 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.028940 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:23.028942 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:23.029040 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.029052 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.029056 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.029126 (+ 70us) tablet.cc:698] Row locks acquired
0110 02:51:23.029128 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.029159 (+ 31us) write_op.cc:270] Start()
0110 02:51:23.029189 (+ 30us) write_op.cc:276] Timestamp: P: 1736477483029100 usec, L: 0
0110 02:51:23.029192 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.029377 (+ 185us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.033038 (+ 3661us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.033169 (+ 131us) write_op.cc:301] APPLY: starting
0110 02:51:23.033181 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.033220 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.033240 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.033342 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.033371 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.033378 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:23.033437 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.033488 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.033502 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:23.033508 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.033573 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.033577 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":395,"apply.run_wall_time_us":395,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":1343,"raft.queue_time_us":56,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":225,"replication_time_us":3815}]]}
I20250110 02:51:23.035876 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_c7ce5ef97f694508890c6699a639c198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:23.036756 1181 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.036888 1181 ts_tablet_manager.cc:1397] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
W20250110 02:51:23.036890 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7ce5ef97f694508890c6699a639c198 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.037669 1181 raft_consensus.cc:357] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.037901 1181 raft_consensus.cc:383] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.037958 1181 raft_consensus.cc:738] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.020658 2240 leader_election.cc:290] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.038157 1181 consensus_queue.cc:260] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.039361 1181 ts_tablet_manager.cc:1428] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:23.040117 1181 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.041674 1184 tablet_bootstrap.cc:654] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.042263 1181 tablet_bootstrap.cc:654] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.042737 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 68) took 1212 ms. Trace:
I20250110 02:51:23.042857 1448 rpcz_store.cc:276] 0110 02:51:21.830368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.027424 (+1197056us) service_pool.cc:225] Handling call
0110 02:51:23.028242 (+ 818us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.028696 (+ 454us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.040648 (+ 11952us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.042619 (+ 1971us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.042694 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.030190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.030282 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.030287 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:23.030290 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:23.030398 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.030406 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.030412 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.030472 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:23.030474 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.030501 (+ 27us) write_op.cc:270] Start()
0110 02:51:23.030527 (+ 26us) write_op.cc:276] Timestamp: P: 1736477483030443 usec, L: 0
0110 02:51:23.030530 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.030653 (+ 123us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.035838 (+ 5185us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.035955 (+ 117us) write_op.cc:301] APPLY: starting
0110 02:51:23.035966 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.036015 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.036048 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.036132 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.036158 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.036164 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:23.036217 (+ 53us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.036269 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.036288 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:23.036296 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.036365 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.036371 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":395,"num_ops":2,"prepare.queue_time_us":1061,"prepare.run_cpu_time_us":563,"prepare.run_wall_time_us":1160,"raft.queue_time_us":32,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":5271}]]}
I20250110 02:51:23.045272 2223 ts_tablet_manager.cc:1939] T 35ec1dba54a247d5ba9f9e053aa3b645 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:22.937809 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6c4bd5e8317a4801a8134d58c23b7bee consensus state has no leader
I20250110 02:51:22.993659 2253 consensus_queue.cc:237] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.046813 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.046813 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.047107 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.048296 2258 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.048795 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.048805 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.050736 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 35ec1dba54a247d5ba9f9e053aa3b645 (table default.Table_6162127f127347419432d61cdac426bf [id=dabf02534dce4d098ef0889ed1359e8e]) successfully deleted
I20250110 02:51:23.051164 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.052393 2240 consensus_queue.cc:1035] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.053424 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 69) took 1189 ms. Trace:
I20250110 02:51:23.055830 1181 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.055945 1181 ts_tablet_manager.cc:1397] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250110 02:51:23.056071 1450 rpcz_store.cc:276] 0110 02:51:21.863928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.035675 (+1171747us) service_pool.cc:225] Handling call
0110 02:51:23.036967 (+ 1292us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.037359 (+ 392us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.053119 (+ 15760us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.053126 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:23.053213 (+ 87us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.053310 (+ 97us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.053395 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.041477 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.041777 (+ 300us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.041784 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:23.041786 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:23.041860 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.041872 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.041876 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.041934 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:23.041936 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.041964 (+ 28us) write_op.cc:270] Start()
0110 02:51:23.042005 (+ 41us) write_op.cc:276] Timestamp: P: 1736477483041904 usec, L: 0
0110 02:51:23.042008 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.042179 (+ 171us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.045300 (+ 3121us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.045461 (+ 161us) write_op.cc:301] APPLY: starting
0110 02:51:23.045473 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.045508 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.045537 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.045620 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.045645 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.045651 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:23.045711 (+ 60us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.045787 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.045812 (+ 25us) write_op.cc:454] Released schema lock
0110 02:51:23.045818 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":401,"num_ops":2,"prepare.queue_time_us":3702,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":6040,"raft.queue_time_us":142,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":265,"replication_time_us":3270,"spinlock_wait_cycles":16768}]]}
W20250110 02:51:23.056738 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d424b4da66184bd19eccefad80621516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d424b4da66184bd19eccefad80621516 already in progress: creating tablet
I20250110 02:51:23.057314 1181 raft_consensus.cc:357] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.058039 1181 raft_consensus.cc:383] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.058101 1181 raft_consensus.cc:738] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.057914 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_e8505c78e8614d1b9424cf824a59dbb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:23.058802 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8505c78e8614d1b9424cf824a59dbb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.058706 1181 consensus_queue.cc:260] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.060181 1181 ts_tablet_manager.cc:1428] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:23.060734 1181 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.062037 2055 consensus_queue.cc:1035] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.062700 1181 tablet_bootstrap.cc:654] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.070719 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 71) took 1197 ms. Trace:
I20250110 02:51:23.071542 1450 rpcz_store.cc:276] 0110 02:51:21.873184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.057695 (+1184511us) service_pool.cc:225] Handling call
0110 02:51:23.058931 (+ 1236us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.060683 (+ 1752us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.070432 (+ 9749us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.070437 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:23.070495 (+ 58us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.070604 (+ 109us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.070676 (+ 72us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.065263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.065360 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.065366 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:23.065368 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:23.065453 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.065466 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.065471 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.065538 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:23.065540 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.065572 (+ 32us) write_op.cc:270] Start()
0110 02:51:23.065608 (+ 36us) write_op.cc:276] Timestamp: P: 1736477483065508 usec, L: 0
0110 02:51:23.065612 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.065765 (+ 153us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.068219 (+ 2454us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.068331 (+ 112us) write_op.cc:301] APPLY: starting
0110 02:51:23.068344 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.068386 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.068408 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.068511 (+ 103us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.068543 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.068550 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:23.068613 (+ 63us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.068667 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.068686 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:23.068703 (+ 17us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":416,"num_ops":2,"prepare.queue_time_us":4051,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":599,"raft.queue_time_us":131,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":208,"replication_time_us":2582}]]}
I20250110 02:51:23.025766 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:23.065063 2139 raft_consensus.cc:491] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.073477 2139 raft_consensus.cc:513] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.074433 2240 consensus_queue.cc:1035] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:23.074544 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7564208da99449cd97338a41b263e750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7564208da99449cd97338a41b263e750 already in progress: creating tablet
I20250110 02:51:23.074855 2139 leader_election.cc:290] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.075183 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.075592 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.075881 775 raft_consensus.cc:2388] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:23.076448 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.074823 1188 raft_consensus.cc:383] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.077102 1188 raft_consensus.cc:738] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.077348 1188 consensus_queue.cc:260] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.077828 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.078136 1181 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.078161 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fc9826ef554a5699e365d10d02e97a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:23.078967 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet fa40c9c3f0ef447294b7c1c5f29ba06e consensus state has no leader
I20250110 02:51:23.078359 1181 ts_tablet_manager.cc:1397] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250110 02:51:23.079739 853 leader_election.cc:304] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:23.080011 2139 raft_consensus.cc:3054] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.079923 1181 raft_consensus.cc:357] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.080139 1181 raft_consensus.cc:383] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.080194 1181 raft_consensus.cc:738] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.081292 1181 consensus_queue.cc:260] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.082682 2139 raft_consensus.cc:2743] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.084666 2272 tablet.cc:1721] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250110 02:51:23.081362 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 consensus state has no leader
I20250110 02:51:23.043973 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_3476c4e4e824479d88c7ef9099d31e87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:23.086540 2240 raft_consensus.cc:491] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.086673 2240 raft_consensus.cc:513] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.086866 1181 ts_tablet_manager.cc:1428] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250110 02:51:23.087258 2240 leader_election.cc:290] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.087592 1680 raft_consensus.cc:491] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.087708 1680 raft_consensus.cc:513] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.088204 1680 leader_election.cc:290] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:23.089016 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.089196 916 raft_consensus.cc:2463] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.089407 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.089633 994 leader_election.cc:304] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:23.089710 1181 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.089942 2055 raft_consensus.cc:2798] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.090018 2055 raft_consensus.cc:491] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.090097 2055 raft_consensus.cc:3054] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.091969 1181 tablet_bootstrap.cc:654] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.092211 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.092407 767 raft_consensus.cc:2463] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.092722 2055 raft_consensus.cc:513] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.093117 1188 ts_tablet_manager.cc:1428] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.070s user 0.000s sys 0.002s
I20250110 02:51:23.093181 2055 leader_election.cc:290] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.093396 1188 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:23.095368 1188 tablet_bootstrap.cc:654] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.097275 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:23.097748 994 leader_election.cc:343] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.098461 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.098613 911 raft_consensus.cc:3054] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.101583 911 raft_consensus.cc:2463] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.101953 994 leader_election.cc:304] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:23.104187 2266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 02:51:23.104897 2139 raft_consensus.cc:491] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.105224 2139 raft_consensus.cc:513] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.105708 2240 raft_consensus.cc:2798] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.105877 2240 raft_consensus.cc:695] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.106160 2240 consensus_queue.cc:237] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.106570 2139 leader_election.cc:290] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:23.107205 853 leader_election.cc:343] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:23.109104 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:23.109843 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 598bc151d0c14fc6bb28523c535d9366 consensus state has no leader
I20250110 02:51:23.112506 1990 consensus_queue.cc:1035] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.113795 2002 fs_manager.cc:730] Time spent opening block manager: real 0.131s user 0.005s sys 0.003s
I20250110 02:51:23.114010 2002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
uuid: "8bc934cc708045e9858ee8f8e64660cd"
format_stamp: "Formatted at 2025-01-10 02:51:05 on dist-test-slave-bpn1"
I20250110 02:51:23.115368 2002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 02:51:23.122566 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.122822 776 raft_consensus.cc:2463] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:23.123051 853 leader_election.cc:343] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:23.123179 853 leader_election.cc:304] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:23.123484 996 leader_election.cc:304] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:23.123837 2199 raft_consensus.cc:491] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.124114 2024 raft_consensus.cc:2798] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.124123 2139 raft_consensus.cc:2743] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.124233 2024 raft_consensus.cc:491] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.124567 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa40c9c3f0ef447294b7c1c5f29ba06e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.124686 776 raft_consensus.cc:3054] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.129565 2272 tablet.cc:1721] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250110 02:51:23.134286 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:23.135499 1832 consensus_queue.cc:1035] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.123983 2199 raft_consensus.cc:513] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.136397 2199 leader_election.cc:290] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.142148 776 raft_consensus.cc:2463] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:23.143267 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7e554d00d05b4fa7a14c9b6159f9d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e554d00d05b4fa7a14c9b6159f9d726 already in progress: creating tablet
W20250110 02:51:23.146188 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
W20250110 02:51:23.146452 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7564208da99449cd97338a41b263e750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7564208da99449cd97338a41b263e750 already in progress: creating tablet
W20250110 02:51:23.151695 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 871180863899419595cd632e7a4cf0dc consensus state has no leader
I20250110 02:51:23.153810 2186 consensus_queue.cc:1035] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.154521 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.131371 1990 consensus_queue.cc:1035] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.139824 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.147464 1041 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.147619 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.158398 1059 raft_consensus.cc:2463] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:23.147460 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.147799 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.148255 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.148819 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.149262 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.149593 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:23.155355 709 leader_election.cc:343] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.164132 2024 raft_consensus.cc:3054] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.164191 711 leader_election.cc:304] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:23.165045 2186 raft_consensus.cc:2798] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.165115 2186 raft_consensus.cc:491] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.165169 2186 raft_consensus.cc:3054] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.167793 2024 raft_consensus.cc:513] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.168160 2186 raft_consensus.cc:513] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.169593 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.169842 770 raft_consensus.cc:3054] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.173022 770 raft_consensus.cc:2463] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.173485 2186 leader_election.cc:290] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.173956 996 leader_election.cc:304] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.174589 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:23.174705 1990 raft_consensus.cc:2798] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.174809 1059 raft_consensus.cc:3054] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.152252 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.153155 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.154477 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.155791 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.174830 1990 raft_consensus.cc:695] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.175364 1990 consensus_queue.cc:237] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.178139 1059 raft_consensus.cc:2463] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:23.179488 711 leader_election.cc:304] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.179975 2186 raft_consensus.cc:2798] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.180157 2186 raft_consensus.cc:695] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:23.180505 2186 consensus_queue.cc:237] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.195852 1184 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.198316 1184 ts_tablet_manager.cc:1397] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.342s user 0.004s sys 0.008s
I20250110 02:51:23.199290 1184 raft_consensus.cc:357] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.201143 1184 raft_consensus.cc:383] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.201258 1184 raft_consensus.cc:738] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.201469 2024 leader_election.cc:290] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:23.201686 1184 consensus_queue.cc:260] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.202860 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ae8cc8c2c4ffa9071e23f7d7c5e11" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.202342 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:23.203326 994 leader_election.cc:343] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.203325 709 leader_election.cc:343] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.206578 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3476c4e4e824479d88c7ef9099d31e87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.151000 2224 ts_tablet_manager.cc:1918] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:23.219826 2224 log.cc:1198] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:23.220257 2224 ts_tablet_manager.cc:1939] T d6fd370138ac4039be55642f1b4c5607 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:23.220706 711 consensus_peers.cc:487] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:23.221477 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d6fd370138ac4039be55642f1b4c5607 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]) successfully deleted
W20250110 02:51:23.221773 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4af23720048fe98b3231ba402d45c already in progress: creating tablet
W20250110 02:51:23.222244 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b382d634db9642df8e00c5ccaa5787a9 already in progress: creating tablet
I20250110 02:51:23.223309 1188 tablet_bootstrap.cc:492] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.223430 1188 ts_tablet_manager.cc:1397] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.000s
I20250110 02:51:23.224089 1188 raft_consensus.cc:357] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.224267 1188 raft_consensus.cc:383] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.224309 1188 raft_consensus.cc:738] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.224468 1188 consensus_queue.cc:260] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.224910 1188 ts_tablet_manager.cc:1428] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:23.225179 1188 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:23.226572 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9289de0630804f9aa60fdaf5f3f15f68 already in progress: creating tablet
W20250110 02:51:23.226878 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d970e2fe72b4727968ca605b3e772cc already in progress: creating tablet
W20250110 02:51:23.227073 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9cf980d9c4117b4ec3ca937cfcf01 already in progress: creating tablet
W20250110 02:51:23.227351 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:23.227560 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302af84fd8374ffa96a7503b54849536 already in progress: creating tablet
I20250110 02:51:23.238200 1184 ts_tablet_manager.cc:1428] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250110 02:51:23.240242 1188 tablet_bootstrap.cc:654] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.250633 1184 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.252395 1184 tablet_bootstrap.cc:654] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.255124 2038 consensus_queue.cc:1035] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.258917 1184 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.259034 1184 ts_tablet_manager.cc:1397] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:23.268960 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.270028 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.271282 2253 raft_consensus.cc:491] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.271413 2253 raft_consensus.cc:513] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.271859 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 70) took 1398 ms. Trace:
I20250110 02:51:23.271961 2253 leader_election.cc:290] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.272010 1448 rpcz_store.cc:276] 0110 02:51:21.873065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.043783 (+1170718us) service_pool.cc:225] Handling call
0110 02:51:23.261660 (+217877us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.262256 (+ 596us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.271534 (+ 9278us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.271662 (+ 128us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.271816 (+ 154us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.266189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.266323 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.266333 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:23.266337 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:23.266486 (+ 149us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.266502 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.266509 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.266620 (+ 111us) tablet.cc:698] Row locks acquired
0110 02:51:23.266624 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:23.266662 (+ 38us) write_op.cc:270] Start()
0110 02:51:23.266702 (+ 40us) write_op.cc:276] Timestamp: P: 1736477483266625 usec, L: 0
0110 02:51:23.266707 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.266904 (+ 197us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.270414 (+ 3510us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.270564 (+ 150us) write_op.cc:301] APPLY: starting
0110 02:51:23.270578 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.270623 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.270650 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.270756 (+ 106us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.270802 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.270813 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:23.270906 (+ 93us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.270971 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.270996 (+ 25us) write_op.cc:454] Released schema lock
0110 02:51:23.271004 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.271095 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.271099 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":536,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":856,"prepare.run_wall_time_us":856,"raft.queue_time_us":207,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":306,"replication_time_us":3661,"spinlock_wait_cycles":116608}]]}
I20250110 02:51:23.272552 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.273358 709 leader_election.cc:343] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.273162 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:23.273895 711 leader_election.cc:343] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.273995 711 leader_election.cc:304] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.274324 2253 raft_consensus.cc:2743] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.274299 1184 raft_consensus.cc:357] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.274494 1184 raft_consensus.cc:383] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.274551 1184 raft_consensus.cc:738] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.274729 1184 consensus_queue.cc:260] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.277710 2258 tablet.cc:1721] T 60486a3dbc5a48eaa26735b84ae3fd6f P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:23.278245 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8c7fbdab52014dddbf63cfef69b09b7b consensus state has no leader
I20250110 02:51:23.278285 2257 tablet.cc:1721] T 60486a3dbc5a48eaa26735b84ae3fd6f P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:23.281368 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e9abd7180bf64bb5991be5f67ea42fbf consensus state has no leader
I20250110 02:51:23.282135 1188 tablet_bootstrap.cc:492] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.282310 1188 ts_tablet_manager.cc:1397] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.003s
W20250110 02:51:23.282771 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 39cfc508c1b84299a2f16d307bb25354 consensus state has no leader
I20250110 02:51:23.283202 1188 raft_consensus.cc:357] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.283424 1188 raft_consensus.cc:383] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.283491 1188 raft_consensus.cc:738] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.283501 2224 tablet_replica.cc:331] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:23.283741 1188 consensus_queue.cc:260] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.284337 1188 ts_tablet_manager.cc:1428] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:23.284639 1188 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:23.285157 2199 raft_consensus.cc:491] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.285270 2199 raft_consensus.cc:513] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.285862 2199 leader_election.cc:290] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.286306 1188 tablet_bootstrap.cc:654] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.286636 1184 ts_tablet_manager.cc:1428] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250110 02:51:23.286885 1184 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.291607 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_89e924c44eb640448848f5b5960dca76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:23.292145 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89e924c44eb640448848f5b5960dca76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.294478 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.294963 709 leader_election.cc:343] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.296422 1184 tablet_bootstrap.cc:654] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.299963 1188 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.300086 1188 ts_tablet_manager.cc:1397] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250110 02:51:23.301254 1184 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.301900 2272 tablet.cc:1721] T 60486a3dbc5a48eaa26735b84ae3fd6f P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.305485 1188 raft_consensus.cc:357] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.305657 1188 raft_consensus.cc:383] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.305985 1184 ts_tablet_manager.cc:1397] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250110 02:51:23.305701 1188 raft_consensus.cc:738] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.306682 1188 consensus_queue.cc:260] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.307269 1188 ts_tablet_manager.cc:1428] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250110 02:51:23.307559 1188 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:23.314682 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:23.315668 1188 tablet_bootstrap.cc:654] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.319458 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:23.319628 1188 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.319768 1188 ts_tablet_manager.cc:1397] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20250110 02:51:23.319960 2257 tablet.cc:1721] T 0b91f351e02d406f8675eb44c4c04ced P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.320048 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:23.320366 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:23.320538 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:23.320502 1188 raft_consensus.cc:357] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.320669 1188 raft_consensus.cc:383] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.320724 1188 raft_consensus.cc:738] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.320719 1441 catalog_manager.cc:5526] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:23.320881 1188 consensus_queue.cc:260] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.321157 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:23.321395 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:23.321576 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:23.321770 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:23.322086 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:23.322248 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:23.322405 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:23.322557 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09]). Expected version 4 got 3
I20250110 02:51:23.322742 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:23.323031 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:23.323182 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:23.323585 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:23.323699 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:23.324375 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:23.324522 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:23.324815 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:23.325075 1441 catalog_manager.cc:5526] T 6e0588e5e5714cc5bd1c73665e5645b1 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:23.325531 1441 catalog_manager.cc:5526] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:23.325839 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:23.327895 1188 ts_tablet_manager.cc:1428] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250110 02:51:23.328203 1188 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:23.330112 1188 tablet_bootstrap.cc:654] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:23.337385 2287 log.cc:926] Time spent T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250110 02:51:23.337934 1184 raft_consensus.cc:357] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.340780 1184 raft_consensus.cc:383] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.340839 1184 raft_consensus.cc:738] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.341027 1184 consensus_queue.cc:260] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.343592 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 72) took 1433 ms. Trace:
W20250110 02:51:23.344553 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7064f3f16f6144a4b624eff2ad3ccf62 consensus state has no leader
I20250110 02:51:23.345767 1184 ts_tablet_manager.cc:1428] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
W20250110 02:51:23.346186 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet deaa3527aa074e6ba94960c672aee028 consensus state has no leader
I20250110 02:51:23.347916 1184 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.344023 1448 rpcz_store.cc:276] 0110 02:51:21.910562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.291333 (+1380771us) service_pool.cc:225] Handling call
0110 02:51:23.292231 (+ 898us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.292711 (+ 480us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.343221 (+ 50510us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.343230 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:23.343320 (+ 90us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.343452 (+ 132us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.343550 (+ 98us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.293320 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.293404 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.293410 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:23.293412 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:23.295016 (+ 1604us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.295055 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.295060 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.295141 (+ 81us) tablet.cc:698] Row locks acquired
0110 02:51:23.295145 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:23.295184 (+ 39us) write_op.cc:270] Start()
0110 02:51:23.295222 (+ 38us) write_op.cc:276] Timestamp: P: 1736477483295146 usec, L: 0
0110 02:51:23.295226 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.295416 (+ 190us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.298619 (+ 3203us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.298811 (+ 192us) write_op.cc:301] APPLY: starting
0110 02:51:23.298826 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.298866 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.298888 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.298984 (+ 96us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.299017 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.299025 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:23.299109 (+ 84us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.299161 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.299180 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:23.299186 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":105,"apply.run_cpu_time_us":441,"apply.run_wall_time_us":441,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":2204,"raft.queue_time_us":169,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":184,"replication_time_us":3366}]]}
I20250110 02:51:23.350129 2272 tablet.cc:1721] T 0b91f351e02d406f8675eb44c4c04ced P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.354856 1188 tablet_bootstrap.cc:492] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:23.358233 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 7e554d00d05b4fa7a14c9b6159f9d726 consensus state has no leader
I20250110 02:51:23.359699 1188 ts_tablet_manager.cc:1397] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250110 02:51:23.360585 1188 raft_consensus.cc:357] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.360848 1188 raft_consensus.cc:383] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.360913 1188 raft_consensus.cc:738] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.361115 1188 consensus_queue.cc:260] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.361613 1188 ts_tablet_manager.cc:1428] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:23.368683 1188 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:23.370229 1188 tablet_bootstrap.cc:654] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.370608 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:23.370916 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:23.371466 1184 tablet_bootstrap.cc:654] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.371878 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:23.372462 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:23.374814 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.375291 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.375980 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.376232 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:23.377490 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.377665 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.377676 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:23.377923 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.377974 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:23.378971 2224 raft_consensus.cc:2238] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:23.379407 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.379999 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.380157 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.380263 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:23.380306 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.380411 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:23.380430 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.380620 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:23.380823 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:23.381174 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:23.381390 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.381409 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:23.381564 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:23.381901 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:23.382211 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:23.382623 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:23.383065 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:23.383275 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.383327 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:23.383603 2224 pending_rounds.cc:62] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:23.383663 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:23.383899 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:23.384235 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
W20250110 02:51:23.384356 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bb595013e809480a814cb9c7aed856e6 consensus state has no leader
W20250110 02:51:23.384752 711 leader_election.cc:343] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.384872 711 leader_election.cc:304] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.384953 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.385033 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 6 got 5
I20250110 02:51:23.385190 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
W20250110 02:51:23.385234 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4bd2283e7ed7456dae466a8582587eb7 consensus state has no leader
W20250110 02:51:23.385449 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet eed058931194480ca1a27130bfa8ee32 consensus state has no leader
I20250110 02:51:23.386071 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
W20250110 02:51:23.386125 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 467f87955c4a4d27a6f0483aa109b922 consensus state has no leader
I20250110 02:51:23.386212 2139 consensus_queue.cc:1035] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.387097 2224 pending_rounds.cc:69] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611768499687424 op_type: NO_OP noop_request { }
I20250110 02:51:23.387245 2224 raft_consensus.cc:2883] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:23.387352 2224 raft_consensus.cc:2267] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.387555 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.387763 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.391132 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.391526 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.391618 1181 tablet_bootstrap.cc:492] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.392128 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.392271 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.392356 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.394493 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:23.394732 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:23.394923 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:23.395529 1181 ts_tablet_manager.cc:1397] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.302s user 0.032s sys 0.054s
I20250110 02:51:23.395673 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.395767 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.396518 2224 ts_tablet_manager.cc:1905] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:23.400800 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.400919 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.401337 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.403988 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.406512 2258 tablet.cc:1721] T 0b91f351e02d406f8675eb44c4c04ced P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.407377 1680 raft_consensus.cc:491] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.407516 1680 raft_consensus.cc:513] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.408655 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.408865 770 raft_consensus.cc:2463] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.409519 996 leader_election.cc:304] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.409761 1680 leader_election.cc:290] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.410571 2289 raft_consensus.cc:2798] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.410670 2289 raft_consensus.cc:491] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.410933 2289 raft_consensus.cc:3054] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.413559 2289 raft_consensus.cc:513] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.415608 2289 leader_election.cc:290] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.415652 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.415817 770 raft_consensus.cc:3054] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.420349 770 raft_consensus.cc:2463] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:23.420877 1434 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 consensus state has no leader
I20250110 02:51:23.426731 2199 raft_consensus.cc:2743] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.430727 2289 raft_consensus.cc:491] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.430685 1181 raft_consensus.cc:357] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.430701 996 leader_election.cc:304] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.434176 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.434505 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff9b10c419242cf85ef1b708bd5a862" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.439698 994 leader_election.cc:343] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.442929 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 235ffe7bf29a4cbcb4faef6f3b2931ec consensus state has no leader
W20250110 02:51:23.443506 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 04a4e7142e83493b98f71487df3f304d consensus state has no leader
I20250110 02:51:23.443562 2139 raft_consensus.cc:491] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.443662 2139 raft_consensus.cc:513] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.444219 2139 leader_election.cc:290] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:23.444401 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3e4d2bb0b992474d8e5d21abaca5411c consensus state has no leader
W20250110 02:51:23.444607 1440 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 09fd9734714a4847ab2a4ebc37af8960 consensus state has no leader
I20250110 02:51:23.445075 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.445281 770 raft_consensus.cc:2388] T 7b3d6e0c043449688ed6450cb2a981a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:23.445742 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.445885 2289 raft_consensus.cc:513] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.450238 1680 raft_consensus.cc:2798] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.451133 1181 raft_consensus.cc:383] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.451304 1181 raft_consensus.cc:738] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.451607 1181 consensus_queue.cc:260] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.452546 1181 ts_tablet_manager.cc:1428] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20250110 02:51:23.452760 1184 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.452903 1184 ts_tablet_manager.cc:1397] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.008s
I20250110 02:51:23.453274 1181 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.454190 1184 raft_consensus.cc:357] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.454466 1184 raft_consensus.cc:383] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.454524 1184 raft_consensus.cc:738] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.454761 1184 consensus_queue.cc:260] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:23.456403 994 leader_election.cc:343] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.459017 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.459184 767 raft_consensus.cc:2463] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.460005 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.460312 1680 raft_consensus.cc:695] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.460669 1680 consensus_queue.cc:237] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.462145 2289 leader_election.cc:290] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.462344 996 leader_election.cc:304] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.462662 2289 raft_consensus.cc:2798] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.462754 2289 raft_consensus.cc:491] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.462877 2289 raft_consensus.cc:3054] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.465591 2289 raft_consensus.cc:513] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.466471 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.466625 767 raft_consensus.cc:3054] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.469209 767 raft_consensus.cc:2463] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.469552 2139 raft_consensus.cc:491] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.469693 2139 raft_consensus.cc:513] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.470234 2139 leader_election.cc:290] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.470722 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.472663 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.472816 767 raft_consensus.cc:2463] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:23.473928 853 leader_election.cc:304] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.473979 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.474494 2139 raft_consensus.cc:2798] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.474582 2139 raft_consensus.cc:491] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.474634 2139 raft_consensus.cc:3054] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.474709 2224 ts_tablet_manager.cc:1918] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:23.474838 2224 log.cc:1198] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:23.475867 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.476014 2224 ts_tablet_manager.cc:1939] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:23.477319 2224 tablet_replica.cc:331] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:23.477257 2139 raft_consensus.cc:513] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.477567 2224 raft_consensus.cc:2238] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:23.477772 2224 raft_consensus.cc:2267] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.477990 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.478907 2139 leader_election.cc:290] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.479290 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.479475 767 raft_consensus.cc:3054] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.482744 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.482975 2289 leader_election.cc:290] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.475237 996 leader_election.cc:304] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.483582 1990 raft_consensus.cc:2798] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.483649 1184 ts_tablet_manager.cc:1428] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250110 02:51:23.483677 1990 raft_consensus.cc:695] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.483889 1990 consensus_queue.cc:237] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.484103 1184 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.484959 2024 consensus_queue.cc:1035] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.485605 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.489306 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:23.489588 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:23.489688 912 raft_consensus.cc:3054] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.489941 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.493033 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.489936 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:23.494500 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.494978 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
W20250110 02:51:23.496768 1440 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 115) took 3300 ms (client timeout 1000 ms). Trace:
I20250110 02:51:23.497174 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:23.497737 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:23.498407 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:23.499085 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:23.493630 2024 raft_consensus.cc:491] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.505817 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.506253 1184 tablet_bootstrap.cc:654] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.507220 2024 raft_consensus.cc:513] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.508170 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.509110 2024 leader_election.cc:290] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:23.509306 994 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.509508 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.509727 772 raft_consensus.cc:2463] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.509718 2024 consensus_queue.cc:1035] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.510052 996 leader_election.cc:304] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:23.510192 994 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.510274 2024 raft_consensus.cc:2798] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.510329 2024 raft_consensus.cc:491] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.510376 2024 raft_consensus.cc:3054] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.514436 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.514602 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.514714 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.514832 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.516690 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:23.519826 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:23.520900 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.521076 914 raft_consensus.cc:2463] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.521378 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:23.483098 767 raft_consensus.cc:2463] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:23.524607 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:23.525151 853 leader_election.cc:304] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.525481 2139 raft_consensus.cc:2798] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.525730 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:23.525760 2139 raft_consensus.cc:695] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:23.526067 2139 consensus_queue.cc:237] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.526923 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:23.527874 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:23.528685 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:23.529364 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:23.530040 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:23.530997 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:23.531948 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09]). Expected version 4 got 3
I20250110 02:51:23.532902 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:23.538262 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:23.538533 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:23.538861 2199 consensus_queue.cc:1035] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.542237 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:23.543452 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:23.544063 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:23.544945 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:23.545188 2258 tablet.cc:1721] T b3a9eaf548274b73a5652ba314dab70c P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.546331 1184 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.546486 1184 ts_tablet_manager.cc:1397] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20250110 02:51:23.547392 1184 raft_consensus.cc:357] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.547607 1184 raft_consensus.cc:383] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.547695 1184 raft_consensus.cc:738] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.547900 1184 consensus_queue.cc:260] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.548750 1184 ts_tablet_manager.cc:1428] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:23.549047 1184 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.549119 2024 raft_consensus.cc:513] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.549669 2024 leader_election.cc:290] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.549520 2306 tablet.cc:1721] T 3934e6611d4e4b94ba5185f33c3d05f4 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.550125 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.550297 772 raft_consensus.cc:3054] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.550840 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:23.551424 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:23.551625 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
W20250110 02:51:23.496925 1440 rpcz_store.cc:269] 0110 02:51:20.196066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.268284 (+ 72218us) service_pool.cc:225] Handling call
0110 02:51:20.294560 (+ 26276us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538df66b8 00007f952fefb716 00007f952fefb445 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:20.672600 (+378040us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.672609 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:22.624188 (+1951579us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.624195 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:23.496719 (+872524us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:20.662011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:20.662165 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:20.662174 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:20.662178 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:20.662524 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:20.662542 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:20.662549 (+ 7us) tablet.cc:682] Acquiring locks for 11 operations
0110 02:51:20.662766 (+ 217us) tablet.cc:698] Row locks acquired
0110 02:51:20.662768 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:20.662817 (+ 49us) write_op.cc:270] Start()
0110 02:51:20.662852 (+ 35us) write_op.cc:276] Timestamp: P: 1736477480662790 usec, L: 0
0110 02:51:20.662855 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:20.663060 (+ 205us) log.cc:844] Serialized 4268 byte log entry
0110 02:51:20.669860 (+ 6800us) op_driver.cc:464] REPLICATION: finished
0110 02:51:20.669981 (+ 121us) write_op.cc:301] APPLY: starting
0110 02:51:20.669996 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:20.670053 (+ 57us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:20.670082 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:20.670228 (+ 146us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:20.670264 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0110 02:51:20.670270 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:20.670354 (+ 84us) log.cc:844] Serialized 83 byte log entry
0110 02:51:20.670426 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:20.670454 (+ 28us) write_op.cc:454] Released schema lock
0110 02:51:20.670461 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:22.581154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.581280 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.581287 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:22.581289 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.581392 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.581409 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.581414 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.581475 (+ 61us) tablet.cc:698] Row locks acquired
0110 02:51:22.581477 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.581510 (+ 33us) write_op.cc:270] Start()
0110 02:51:22.581542 (+ 32us) write_op.cc:276] Timestamp: P: 1736477482581450 usec, L: 0
0110 02:51:22.581546 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.581712 (+ 166us) log.cc:844] Serialized 356 byte log entry
0110 02:51:22.596217 (+ 14505us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.596454 (+ 237us) write_op.cc:301] APPLY: starting
0110 02:51:22.596470 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.596511 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.596541 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.596592 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.596623 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.596630 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.596704 (+ 74us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.596758 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.596779 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:22.596785 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.596847 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.596851 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:22.602133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.602266 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.602271 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:22.602273 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.602388 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.602400 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.602405 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.602463 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:22.602465 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.602495 (+ 30us) write_op.cc:270] Start()
0110 02:51:22.602522 (+ 27us) write_op.cc:276] Timestamp: P: 1736477482602478 usec, L: 0
0110 02:51:22.602525 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.602693 (+ 168us) log.cc:844] Serialized 356 byte log entry
0110 02:51:22.606656 (+ 3963us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.606789 (+ 133us) write_op.cc:301] APPLY: starting
0110 02:51:22.606802 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.606845 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.606868 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.606912 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.606939 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.606946 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.607007 (+ 61us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.607056 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.607072 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:22.607077 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.607145 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.607152 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:22.611456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.611573 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.611578 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:22.611579 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:22.611668 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.611678 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.611683 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.611731 (+ 48us) tablet.cc:698] Row locks acquired
0110 02:51:22.611733 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.611759 (+ 26us) write_op.cc:270] Start()
0110 02:51:22.611781 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482611742 usec, L: 0
0110 02:51:22.611784 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.611936 (+ 152us) log.cc:844] Serialized 356 byte log entry
0110 02:51:22.620410 (+ 8474us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.620544 (+ 134us) write_op.cc:301] APPLY: starting
0110 02:51:22.620557 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.620592 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.620626 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.620669 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.620695 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.620702 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.620764 (+ 62us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.620813 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.620842 (+ 29us) write_op.cc:454] Released schema lock
0110 02:51:22.620848 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":24142,"reported_tablets":238,"spinlock_wait_cycles":43953024,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":528,"apply.run_wall_time_us":528,"num_ops":11,"prepare.queue_time_us":1510,"prepare.run_cpu_time_us":1190,"prepare.run_wall_time_us":1195,"raft.queue_time_us":947,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":264,"replication_time_us":6981}],["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":387,"num_ops":1,"prepare.queue_time_us":170,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":656,"raft.queue_time_us":808,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":245,"replication_time_us":14641}],["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":347,"apply.run_wall_time_us":347,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":643,"raft.queue_time_us":129,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":199,"replication_time_us":4104}],["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":350,"apply.run_wall_time_us":350,"num_ops":1,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":572,"raft.queue_time_us":6194,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":191,"replication_time_us":8601}]]}
I20250110 02:51:23.553687 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:23.553893 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_c091cf6d870b4996ae16024d22e7a450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:23.554803 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c091cf6d870b4996ae16024d22e7a450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.554185 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:23.555763 2138 raft_consensus.cc:491] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.555590 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:23.557225 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:23.557811 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:23.558923 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:23.559929 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:23.560813 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:23.561662 2199 raft_consensus.cc:491] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.555894 2138 raft_consensus.cc:513] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.561801 2199 raft_consensus.cc:513] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.562265 2138 leader_election.cc:290] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.562639 2289 consensus_queue.cc:1035] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20250110 02:51:23.564507 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b382d634db9642df8e00c5ccaa5787a9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.564980 709 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.567693 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.568341 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:23.569201 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.569667 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:23.570289 1181 tablet_bootstrap.cc:654] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.570348 896 tablet_service.cc:1467] Processing CreateTablet for tablet b05b16162598487e8dbad300724abdf3 (DEFAULT_TABLE table=default.Table_b1de670e3fe347108cccbf69ab116d85 [id=9d64a6bba43244ab9cd4d445b5e673c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.570451 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d]). Expected version 6 got 5
I20250110 02:51:23.570724 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:23.572634 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.573087 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.573304 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.573470 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.575968 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b05b16162598487e8dbad300724abdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.576602 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.582641 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.582741 2199 leader_election.cc:290] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.582991 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.583290 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.583398 1440 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 73) took 1646 ms. Trace:
I20250110 02:51:23.583647 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.583544 1440 rpcz_store.cc:276] 0110 02:51:21.937353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.553578 (+1616225us) service_pool.cc:225] Handling call
0110 02:51:23.569224 (+ 15646us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.569707 (+ 483us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.576202 (+ 6495us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.583029 (+ 6827us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.583357 (+ 328us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.571057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.571183 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.571192 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:23.571204 (+ 12us) tablet.cc:659] Decoding operations
0110 02:51:23.571315 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.571332 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.571339 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.571441 (+ 102us) tablet.cc:698] Row locks acquired
0110 02:51:23.571445 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:23.571482 (+ 37us) write_op.cc:270] Start()
0110 02:51:23.571519 (+ 37us) write_op.cc:276] Timestamp: P: 1736477483571416 usec, L: 0
0110 02:51:23.571525 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.571755 (+ 230us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.574542 (+ 2787us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.574681 (+ 139us) write_op.cc:301] APPLY: starting
0110 02:51:23.574699 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.574745 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.574770 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.574895 (+ 125us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.574939 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.574946 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:23.575022 (+ 76us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.575083 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.575101 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:23.575107 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.575164 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.575167 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":478,"apply.run_wall_time_us":477,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":835,"prepare.run_wall_time_us":834,"raft.queue_time_us":160,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":236,"replication_time_us":2986}]]}
I20250110 02:51:23.584805 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.585300 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.585572 2272 tablet.cc:1721] T b3a9eaf548274b73a5652ba314dab70c P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.586164 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.587779 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b05b16162598487e8dbad300724abdf3 (DEFAULT_TABLE table=default.Table_b1de670e3fe347108cccbf69ab116d85 [id=9d64a6bba43244ab9cd4d445b5e673c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.588116 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b05b16162598487e8dbad300724abdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.588650 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_cf24cfbc9d274002aa22a9e06810abd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:23.589342 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf24cfbc9d274002aa22a9e06810abd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:23.592191 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (table default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]) successfully deleted
W20250110 02:51:23.592542 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7564208da99449cd97338a41b263e750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7564208da99449cd97338a41b263e750 already in progress: creating tablet
W20250110 02:51:23.592943 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3510e86314144a2ca012aacf3d78eacc already in progress: creating tablet
I20250110 02:51:23.593109 1181 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.593259 1181 ts_tablet_manager.cc:1397] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.140s user 0.006s sys 0.002s
I20250110 02:51:23.593962 1181 raft_consensus.cc:357] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.594238 1181 raft_consensus.cc:383] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.594334 1181 raft_consensus.cc:738] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.594564 1181 consensus_queue.cc:260] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.595597 1832 consensus_queue.cc:1035] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.596017 1181 ts_tablet_manager.cc:1428] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:23.596884 1181 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.597587 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b382d634db9642df8e00c5ccaa5787a9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.598155 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d6e0c043449688ed6450cb2a981a3" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.598624 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.599112 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:23.601243 1181 tablet_bootstrap.cc:654] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.605855 1440 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 74) took 1652 ms. Trace:
I20250110 02:51:23.605973 1440 rpcz_store.cc:276] 0110 02:51:21.953270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.584330 (+1631060us) service_pool.cc:225] Handling call
0110 02:51:23.589448 (+ 5118us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:23.589946 (+ 498us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:23.605064 (+ 15118us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.605070 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:23.605134 (+ 64us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:23.605704 (+ 570us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:23.605817 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.600410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.600564 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.600574 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:23.600578 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:23.600701 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.600719 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.600726 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.600837 (+ 111us) tablet.cc:698] Row locks acquired
0110 02:51:23.600841 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:23.600879 (+ 38us) write_op.cc:270] Start()
0110 02:51:23.600916 (+ 37us) write_op.cc:276] Timestamp: P: 1736477483600809 usec, L: 0
0110 02:51:23.600921 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.601098 (+ 177us) log.cc:844] Serialized 433 byte log entry
0110 02:51:23.604079 (+ 2981us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.604161 (+ 82us) write_op.cc:301] APPLY: starting
0110 02:51:23.604178 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.604219 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.604250 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.604351 (+ 101us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.604384 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:23.604391 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:23.604473 (+ 82us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.604529 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.604548 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:23.604555 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":447,"num_ops":2,"prepare.queue_time_us":9861,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":820,"raft.queue_time_us":185,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":195,"replication_time_us":3125}]]}
I20250110 02:51:23.608518 2024 raft_consensus.cc:491] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.608623 2024 raft_consensus.cc:513] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.609169 2024 leader_election.cc:290] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.610894 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.611589 772 raft_consensus.cc:2463] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
W20250110 02:51:23.611732 711 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.611850 711 leader_election.cc:304] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.615139 2199 raft_consensus.cc:2743] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.609573 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b14d7592024a34baf7756d92e546d9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.617830 918 raft_consensus.cc:3054] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.608386 2257 tablet.cc:1721] T 3934e6611d4e4b94ba5185f33c3d05f4 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.620493 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:23.621428 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:23.623287 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:23.630151 2002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 02:51:23.636638 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:23.639181 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:23.639508 1440 master_service.cc:946] ReplaceTablet: received request to replace tablet 5f229f546a8d4a09b93f8b46c893d14a from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:23.639528 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:23.640426 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:23.640687 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:23.640949 1450 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:23.654853 2138 raft_consensus.cc:491] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.655030 2138 raft_consensus.cc:513] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.656522 2002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 02:51:23.656809 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 02:51:23.636435 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.657619 916 raft_consensus.cc:2463] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.658099 994 leader_election.cc:304] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:23.658433 1849 raft_consensus.cc:2798] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.658537 1849 raft_consensus.cc:491] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.658612 1849 raft_consensus.cc:3054] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.661489 1849 raft_consensus.cc:513] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.663427 1849 leader_election.cc:290] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.667443 2138 leader_election.cc:290] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.668136 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 5f229f546a8d4a09b93f8b46c893d14a
I20250110 02:51:23.669096 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a with delete_type TABLET_DATA_DELETED (replaced by tablet dc6b2654f86e4686b8a3034de02d11bf) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:23.670329 1440 catalog_manager.cc:6596] ReplaceTablet: tablet 5f229f546a8d4a09b93f8b46c893d14a deleted and replaced by tablet dc6b2654f86e4686b8a3034de02d11bf
I20250110 02:51:23.670334 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a with delete_type TABLET_DATA_DELETED (replaced by tablet dc6b2654f86e4686b8a3034de02d11bf) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:23.670576 1440 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 75) took 1493 ms. Trace:
I20250110 02:51:23.670673 1440 rpcz_store.cc:276] 0110 02:51:22.176743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.639412 (+1462669us) service_pool.cc:225] Handling call
0110 02:51:23.667845 (+ 28433us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.667850 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:23.670517 (+ 2667us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.661889 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.662002 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.662009 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:23.662010 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:23.662117 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.662130 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.662134 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:23.662210 (+ 76us) tablet.cc:698] Row locks acquired
0110 02:51:23.662212 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.662247 (+ 35us) write_op.cc:270] Start()
0110 02:51:23.662279 (+ 32us) write_op.cc:276] Timestamp: P: 1736477483662225 usec, L: 0
0110 02:51:23.662282 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.662504 (+ 222us) log.cc:844] Serialized 656 byte log entry
0110 02:51:23.665496 (+ 2992us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.665645 (+ 149us) write_op.cc:301] APPLY: starting
0110 02:51:23.665675 (+ 30us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.665728 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.665764 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.665890 (+ 126us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.665967 (+ 77us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:23.665978 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:23.666061 (+ 83us) log.cc:844] Serialized 29 byte log entry
0110 02:51:23.666121 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.666144 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:23.666152 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":715,"prepare.run_wall_time_us":716,"raft.queue_time_us":139,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":227,"replication_time_us":3181}]]}
I20250110 02:51:23.671767 2352 tablet_replica.cc:331] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:23.672012 2352 raft_consensus.cc:2238] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:23.674301 752 tablet_service.cc:1467] Processing CreateTablet for tablet b05b16162598487e8dbad300724abdf3 (DEFAULT_TABLE table=default.Table_b1de670e3fe347108cccbf69ab116d85 [id=9d64a6bba43244ab9cd4d445b5e673c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.677067 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.677281 771 raft_consensus.cc:2463] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.677649 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a with delete_type TABLET_DATA_DELETED (replaced by tablet dc6b2654f86e4686b8a3034de02d11bf) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:23.678455 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b05b16162598487e8dbad300724abdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.681063 2306 tablet.cc:1721] T 871180863899419595cd632e7a4cf0dc P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.682526 2257 tablet.cc:1721] T a4c6de957fd14af983fc1a3cd8629226 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.685436 2306 tablet.cc:1721] T a4c6de957fd14af983fc1a3cd8629226 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.686290 996 leader_election.cc:304] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.686968 2055 raft_consensus.cc:2798] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.687095 2055 raft_consensus.cc:695] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.687361 2055 consensus_queue.cc:237] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:23.690591 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 84478.4us
I20250110 02:51:23.694954 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.695647 2055 consensus_queue.cc:1035] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.702754 2352 raft_consensus.cc:2267] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.704900 2352 ts_tablet_manager.cc:1905] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:23.673329 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.706034 907 raft_consensus.cc:3054] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.708451 918 raft_consensus.cc:2463] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.712523 1181 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.712749 1181 ts_tablet_manager.cc:1397] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.001s
I20250110 02:51:23.713654 1181 raft_consensus.cc:357] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.713876 1181 raft_consensus.cc:383] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.713923 1181 raft_consensus.cc:738] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.714311 1181 consensus_queue.cc:260] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:23.710830 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
I20250110 02:51:23.716976 853 leader_election.cc:304] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:23.717204 2138 raft_consensus.cc:2743] T 7b3d6e0c043449688ed6450cb2a981a3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:23.717571 853 leader_election.cc:343] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.722270 853 leader_election.cc:343] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.723340 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
I20250110 02:51:23.724506 853 leader_election.cc:304] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:23.724781 2138 raft_consensus.cc:2743] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.729651 2306 tablet.cc:1721] T b6757b7ecc514489a0fa7e1ff18206bd P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.729008 2362 tablet.cc:1721] T 871180863899419595cd632e7a4cf0dc P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.732690 907 raft_consensus.cc:2463] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.733058 994 leader_election.cc:304] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:23.733361 2024 raft_consensus.cc:2798] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.733457 2024 raft_consensus.cc:695] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.733673 2024 consensus_queue.cc:237] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.739579 1188 tablet_bootstrap.cc:492] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.739714 1188 ts_tablet_manager.cc:1397] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.371s user 0.003s sys 0.010s
I20250110 02:51:23.740442 1188 raft_consensus.cc:357] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.740880 1188 raft_consensus.cc:383] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.740942 1188 raft_consensus.cc:738] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.746141 1188 consensus_queue.cc:260] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.746196 2258 tablet.cc:1721] T ec427cc101bd49f6b9d1743e3eeebbd1 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.754364 2272 tablet.cc:1721] T 871180863899419595cd632e7a4cf0dc P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.754793 2253 consensus_queue.cc:1035] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.761346 1990 raft_consensus.cc:491] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.761545 1990 raft_consensus.cc:513] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.762276 1990 leader_election.cc:290] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.767973 2199 raft_consensus.cc:491] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.769263 2258 tablet.cc:1721] T e8b2297377bc4e5095826e30523eede7 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.771050 1184 tablet_bootstrap.cc:654] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.771586 2257 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.775210 2257 tablet.cc:1721] T b6757b7ecc514489a0fa7e1ff18206bd P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.776430 2362 tablet.cc:1721] T b3a9eaf548274b73a5652ba314dab70c P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.782435 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20177229f7b4f11875e0007afb1dd00" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.782743 773 raft_consensus.cc:2388] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:23.782984 2199 raft_consensus.cc:513] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.783762 2199 leader_election.cc:290] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.767805 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd1e726f1a41d0894737b3ef3caf8f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.783999 766 raft_consensus.cc:3054] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.785053 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28340f0f9a2043a199c74f13755eb702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.786193 709 leader_election.cc:343] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.786590 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.786775 765 raft_consensus.cc:2463] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.787456 996 leader_election.cc:304] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.788046 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28340f0f9a2043a199c74f13755eb702" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.789235 2257 tablet.cc:1721] T ec427cc101bd49f6b9d1743e3eeebbd1 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.789316 1990 raft_consensus.cc:2798] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.789438 1990 raft_consensus.cc:491] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.789539 1990 raft_consensus.cc:3054] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.790814 2362 tablet.cc:1721] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.792321 1990 raft_consensus.cc:513] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.793078 2352 ts_tablet_manager.cc:1918] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:23.793229 2352 log.cc:1198] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/5f229f546a8d4a09b93f8b46c893d14a
I20250110 02:51:23.793491 1990 leader_election.cc:290] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.793468 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:23.793906 994 leader_election.cc:343] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.796378 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 70882.2us
I20250110 02:51:23.796552 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:23.796770 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:23.796931 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:23.797101 1181 ts_tablet_manager.cc:1428] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.084s user 0.002s sys 0.000s
I20250110 02:51:23.797179 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:23.797907 1181 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.798888 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.799110 765 raft_consensus.cc:3054] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.800320 1181 tablet_bootstrap.cc:654] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.802183 765 raft_consensus.cc:2463] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.803552 996 leader_election.cc:304] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:23.804046 2024 raft_consensus.cc:2798] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.804737 2024 raft_consensus.cc:695] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.805176 2024 consensus_queue.cc:237] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.797333 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:23.806442 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:23.806612 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:23.806798 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:23.808802 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:23.809041 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:23.809805 766 raft_consensus.cc:2463] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.810088 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:23.810806 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:23.811331 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:23.811558 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:23.811842 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:23.812283 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:23.812441 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:23.812640 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 60486a3dbc5a48eaa26735b84ae3fd6f (table default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39]). Expected version 1 got 0
I20250110 02:51:23.812889 1448 catalog_manager.cc:5526] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:23.813314 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:23.813563 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:23.813763 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:23.814675 2352 ts_tablet_manager.cc:1939] T 5f229f546a8d4a09b93f8b46c893d14a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:23.817469 2272 tablet.cc:1721] T a4c6de957fd14af983fc1a3cd8629226 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.818848 2357 tablet_replica.cc:331] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:23.819093 2357 raft_consensus.cc:2238] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:23.819324 2224 ts_tablet_manager.cc:1905] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:23.820075 711 leader_election.cc:343] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.819348 2357 raft_consensus.cc:2267] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.820875 711 leader_election.cc:304] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.821030 1181 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.821156 1181 ts_tablet_manager.cc:1397] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250110 02:51:23.821792 2253 raft_consensus.cc:2743] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.821781 1181 raft_consensus.cc:357] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.821962 1181 raft_consensus.cc:383] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.822016 1181 raft_consensus.cc:738] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.823264 2357 ts_tablet_manager.cc:1905] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:23.826685 2253 raft_consensus.cc:491] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.826826 2253 raft_consensus.cc:513] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.827684 2253 leader_election.cc:290] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.828090 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452223af5f6a4836a0f582d394a9453e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.828573 1181 consensus_queue.cc:260] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.829084 1181 ts_tablet_manager.cc:1428] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250110 02:51:23.831607 711 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.832403 2366 tablet.cc:1721] T 3934e6611d4e4b94ba5185f33c3d05f4 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.835171 1181 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.836496 1181 tablet_bootstrap.cc:654] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.837373 2258 tablet.cc:1721] T 7902775b8db247a89d9b20d0ee880e36 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.841394 2224 ts_tablet_manager.cc:1918] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:23.850028 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:23.851052 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:23.851449 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:23.851491 2224 log.cc:1198] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/28340f0f9a2043a199c74f13755eb702
I20250110 02:51:23.851740 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:23.851861 2224 ts_tablet_manager.cc:1939] T 28340f0f9a2043a199c74f13755eb702 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:23.852064 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:23.852303 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:23.852632 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:23.853242 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:23.853277 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 28340f0f9a2043a199c74f13755eb702 (table default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]) successfully deleted
I20250110 02:51:23.853626 2224 tablet_replica.cc:331] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:23.853883 2224 raft_consensus.cc:2238] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:23.854086 2224 raft_consensus.cc:2267] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.854964 2224 ts_tablet_manager.cc:1905] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:23.857493 2375 tablet.cc:1721] T b6757b7ecc514489a0fa7e1ff18206bd P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.853432 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:23.860929 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:23.861130 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:23.861580 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:23.861893 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0b91f351e02d406f8675eb44c4c04ced (table default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215]). Expected version 1 got 0
I20250110 02:51:23.862994 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:23.863193 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:23.863391 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:23.863793 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:23.864187 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:23.864615 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:23.864737 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:23.865053 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b3a9eaf548274b73a5652ba314dab70c (table default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326]). Expected version 2 got 0
I20250110 02:51:23.865427 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:23.865648 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3934e6611d4e4b94ba5185f33c3d05f4 (table default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26]). Expected version 1 got 0
I20250110 02:51:23.865849 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:23.867988 2368 tablet.cc:1721] T ec427cc101bd49f6b9d1743e3eeebbd1 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.869589 1181 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.869702 1181 ts_tablet_manager.cc:1397] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20250110 02:51:23.870397 1181 raft_consensus.cc:357] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.870601 1181 raft_consensus.cc:383] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.870651 1181 raft_consensus.cc:738] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.870847 1181 consensus_queue.cc:260] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.871279 1181 ts_tablet_manager.cc:1428] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:23.871546 1181 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:23.873596 1181 tablet_bootstrap.cc:654] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.878459 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:23.878683 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:23.879294 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:23.879639 1448 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:23.883424 1188 ts_tablet_manager.cc:1428] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.144s user 0.001s sys 0.001s
I20250110 02:51:23.883791 1188 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:23.884994 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 88382us
I20250110 02:51:23.885579 1188 tablet_bootstrap.cc:654] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.885679 1832 consensus_queue.cc:1035] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.887250 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]) successfully deleted
I20250110 02:51:23.877149 2375 tablet.cc:1721] T e8b2297377bc4e5095826e30523eede7 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.899066 1184 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.899240 1184 ts_tablet_manager.cc:1397] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.350s user 0.017s sys 0.012s
I20250110 02:51:23.899351 2368 tablet.cc:1721] T 7902775b8db247a89d9b20d0ee880e36 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.900110 1184 raft_consensus.cc:357] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.900391 1184 raft_consensus.cc:383] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.900477 1184 raft_consensus.cc:738] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.900708 1184 consensus_queue.cc:260] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.901381 1184 ts_tablet_manager.cc:1428] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:23.901631 1184 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:23.901844 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452223af5f6a4836a0f582d394a9453e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.902581 709 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.902681 709 leader_election.cc:304] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.903098 2253 raft_consensus.cc:2743] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.903318 1184 tablet_bootstrap.cc:654] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:23.911247 2224 ts_tablet_manager.cc:1918] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:23.911418 2224 log.cc:1198] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:23.911749 2224 ts_tablet_manager.cc:1939] T 8f15078fc22f4b32a48c7190b36e2192 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:23.917809 2253 raft_consensus.cc:491] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.917951 2253 raft_consensus.cc:513] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.918768 2253 leader_election.cc:290] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:23.926496 2357 ts_tablet_manager.cc:1918] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:23.926666 2357 log.cc:1198] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/5f229f546a8d4a09b93f8b46c893d14a
I20250110 02:51:23.927095 2357 ts_tablet_manager.cc:1939] T 5f229f546a8d4a09b93f8b46c893d14a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:23.927593 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.928151 709 leader_election.cc:343] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.928191 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 5f229f546a8d4a09b93f8b46c893d14a (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]) successfully deleted
I20250110 02:51:23.919219 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:23.922286 994 leader_election.cc:343] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:23.932508 711 leader_election.cc:343] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.932610 711 leader_election.cc:304] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.933132 2253 raft_consensus.cc:2743] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.928529 2024 consensus_queue.cc:1035] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.937561 2258 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.941182 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8f15078fc22f4b32a48c7190b36e2192 (table default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]) successfully deleted
I20250110 02:51:23.944573 1184 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:23.944723 1184 ts_tablet_manager.cc:1397] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.009s
I20250110 02:51:23.944578 2359 tablet.cc:1721] T 7902775b8db247a89d9b20d0ee880e36 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:23.945693 1184 raft_consensus.cc:357] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.945919 1184 raft_consensus.cc:383] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.946012 1184 raft_consensus.cc:738] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.947854 1188 tablet_bootstrap.cc:492] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:23.948010 1188 ts_tablet_manager.cc:1397] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.001s
I20250110 02:51:23.948496 896 tablet_service.cc:1467] Processing CreateTablet for tablet d6acae633eb14e5db87b1e36a069422c (DEFAULT_TABLE table=default.Table_e8505c78e8614d1b9424cf824a59dbb9 [id=5b64687362a04c36a23575706bff068d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.948612 1031 tablet_service.cc:1467] Processing CreateTablet for tablet d6acae633eb14e5db87b1e36a069422c (DEFAULT_TABLE table=default.Table_e8505c78e8614d1b9424cf824a59dbb9 [id=5b64687362a04c36a23575706bff068d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.948896 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6acae633eb14e5db87b1e36a069422c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.948827 1188 raft_consensus.cc:357] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.948992 1188 raft_consensus.cc:383] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.949033 1188 raft_consensus.cc:738] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.949687 2253 raft_consensus.cc:491] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.949797 2253 raft_consensus.cc:513] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.950323 2253 leader_election.cc:290] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:23.951706 994 consensus_peers.cc:487] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f229f546a8d4a09b93f8b46c893d14a. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:23.951815 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4af23720048fe98b3231ba402d45c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.952423 709 leader_election.cc:343] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.952486 2038 consensus_queue.cc:1035] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:23.954452 853 leader_election.cc:304] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:23.956279 751 tablet_service.cc:1467] Processing CreateTablet for tablet d6acae633eb14e5db87b1e36a069422c (DEFAULT_TABLE table=default.Table_e8505c78e8614d1b9424cf824a59dbb9 [id=5b64687362a04c36a23575706bff068d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.956806 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6acae633eb14e5db87b1e36a069422c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.957029 1184 consensus_queue.cc:260] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.957191 2038 raft_consensus.cc:3054] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.957597 1184 ts_tablet_manager.cc:1428] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250110 02:51:23.959563 2038 raft_consensus.cc:2743] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.960098 1059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4af23720048fe98b3231ba402d45c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:23.960654 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6acae633eb14e5db87b1e36a069422c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:23.963182 711 leader_election.cc:343] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.963274 711 leader_election.cc:304] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:23.963502 2199 raft_consensus.cc:2743] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:23.964008 896 tablet_service.cc:1467] Processing CreateTablet for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3 (DEFAULT_TABLE table=default.Table_3476c4e4e824479d88c7ef9099d31e87 [id=015b08a56b704be6bd71439b78b33029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.964167 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3 (DEFAULT_TABLE table=default.Table_3476c4e4e824479d88c7ef9099d31e87 [id=015b08a56b704be6bd71439b78b33029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.964320 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.964500 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.965483 2024 raft_consensus.cc:491] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:23.965582 2024 raft_consensus.cc:513] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.966557 2024 leader_election.cc:290] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.966943 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:23.967136 767 raft_consensus.cc:2463] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:23.967489 996 leader_election.cc:304] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.967715 2024 raft_consensus.cc:2798] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:23.967779 2024 raft_consensus.cc:491] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:23.967839 2024 raft_consensus.cc:3054] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.968608 2224 tablet_replica.cc:331] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:23.969982 2224 raft_consensus.cc:2238] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:23.970464 2359 tablet.cc:1721] T 820975aa259f4274ba6c19398425e94b P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:23.970803 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 already in progress: creating tablet
I20250110 02:51:23.969949 2024 raft_consensus.cc:513] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:23.971030 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:23.971935 2024 leader_election.cc:290] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:23.972198 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:23.972375 764 raft_consensus.cc:3054] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:23.972410 1181 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:23.972524 1181 ts_tablet_manager.cc:1397] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.004s
I20250110 02:51:23.973202 1181 raft_consensus.cc:357] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:23.973388 1181 raft_consensus.cc:383] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:23.973446 1181 raft_consensus.cc:738] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:23.975623 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3 (DEFAULT_TABLE table=default.Table_3476c4e4e824479d88c7ef9099d31e87 [id=015b08a56b704be6bd71439b78b33029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.975732 764 raft_consensus.cc:2463] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:23.976063 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c22e4c9b4a04b72b9a3d1ce3c1df5b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.976241 996 leader_election.cc:304] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:23.976706 2024 raft_consensus.cc:2798] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:23.977133 2024 raft_consensus.cc:695] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:23.977483 2024 consensus_queue.cc:237] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.977850 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:23.978989 994 leader_election.cc:343] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.981415 896 tablet_service.cc:1467] Processing CreateTablet for tablet b9275f42fdf24464b580cda000bf8e25 (DEFAULT_TABLE table=default.Table_f3ea92dbdb6e4f3dbc723ffac6b602a1 [id=7bab42d6d35e46b6847ce0bc7c9945fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.981604 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b9275f42fdf24464b580cda000bf8e25 (DEFAULT_TABLE table=default.Table_f3ea92dbdb6e4f3dbc723ffac6b602a1 [id=7bab42d6d35e46b6847ce0bc7c9945fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.981782 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9275f42fdf24464b580cda000bf8e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.981999 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9275f42fdf24464b580cda000bf8e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.983706 2224 pending_rounds.cc:62] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:23.983755 2224 pending_rounds.cc:69] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611773668818944 op_type: NO_OP noop_request { }
I20250110 02:51:23.983836 2224 raft_consensus.cc:2883] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:23.983903 2224 raft_consensus.cc:2267] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:23.984216 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:23.984616 2224 ts_tablet_manager.cc:1905] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:23.985069 994 leader_election.cc:343] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:23.986124 2366 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250110 02:51:23.987515 2375 tablet.cc:1721] T 820975aa259f4274ba6c19398425e94b P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.970357 1188 consensus_queue.cc:260] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:23.991503 1188 ts_tablet_manager.cc:1428] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.043s user 0.001s sys 0.002s
I20250110 02:51:23.993803 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:23.994000 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:23.994131 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:23.994398 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:23.994452 2258 tablet.cc:1721] T 820975aa259f4274ba6c19398425e94b P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:23.994460 752 tablet_service.cc:1467] Processing CreateTablet for tablet b9275f42fdf24464b580cda000bf8e25 (DEFAULT_TABLE table=default.Table_f3ea92dbdb6e4f3dbc723ffac6b602a1 [id=7bab42d6d35e46b6847ce0bc7c9945fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:23.994535 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:23.994704 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet a4c6de957fd14af983fc1a3cd8629226 (table default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b]). Expected version 1 got 0
I20250110 02:51:23.994794 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9275f42fdf24464b580cda000bf8e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:23.994871 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:23.995033 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:23.995200 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:23.995471 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:23.995649 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:23.995822 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:23.996502 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]). Expected version 1 got 0
I20250110 02:51:23.996843 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:23.997031 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:23.997519 1440 catalog_manager.cc:5526] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:23.998301 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:23.998494 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:23.998908 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:23.999329 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:24.000025 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 871180863899419595cd632e7a4cf0dc (table default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d]). Expected version 1 got 0
I20250110 02:51:24.001188 896 tablet_service.cc:1467] Processing CreateTablet for tablet 6598311f59c6476ba339b8e57961cf67 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.001288 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6598311f59c6476ba339b8e57961cf67 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.001516 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6598311f59c6476ba339b8e57961cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.001598 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6598311f59c6476ba339b8e57961cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.002997 1188 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.004597 1188 tablet_bootstrap.cc:654] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.005517 1184 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.006217 2289 consensus_queue.cc:1035] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.006731 1181 consensus_queue.cc:260] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.006851 1184 tablet_bootstrap.cc:654] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.008039 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6598311f59c6476ba339b8e57961cf67 (DEFAULT_TABLE table=default.Table_18a8f14e969a458599b59ee194888979 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.008409 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6598311f59c6476ba339b8e57961cf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.009541 1181 ts_tablet_manager.cc:1428] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250110 02:51:24.010279 1181 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:24.012182 1181 tablet_bootstrap.cc:654] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.013757 913 raft_consensus.cc:3054] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.019999 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89478577a1428e8957879d6d52be28 (DEFAULT_TABLE table=default.Table_119ec4ba01fd4c019b13cd28b0a3b82c [id=e2d675e7ff0244c08b1afe3d26070a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.020385 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89478577a1428e8957879d6d52be28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.021077 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89478577a1428e8957879d6d52be28 (DEFAULT_TABLE table=default.Table_119ec4ba01fd4c019b13cd28b0a3b82c [id=e2d675e7ff0244c08b1afe3d26070a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.021358 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89478577a1428e8957879d6d52be28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.000361 1440 catalog_manager.cc:5526] T 7b3d6e0c043449688ed6450cb2a981a3 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:24.022421 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 820975aa259f4274ba6c19398425e94b (table default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb]). Expected version 1 got 0
I20250110 02:51:24.022606 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_e49b14cd5bac4073be1ad6ffe44eb92c [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 4 got 0
I20250110 02:51:24.022874 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7902775b8db247a89d9b20d0ee880e36 (table default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0]). Expected version 2 got 0
I20250110 02:51:24.023402 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:24.023588 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:24.024281 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:24.024890 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:24.025218 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:24.025362 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:24.027097 892 tablet_service.cc:1467] Processing CreateTablet for tablet 00aa9ad304134509870dd5b276a66fd8 (DEFAULT_TABLE table=default.Table_89e924c44eb640448848f5b5960dca76 [id=f6d0c22dccd041e8b32d343e62c986c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.027489 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aa9ad304134509870dd5b276a66fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.027755 747 tablet_service.cc:1467] Processing CreateTablet for tablet 00aa9ad304134509870dd5b276a66fd8 (DEFAULT_TABLE table=default.Table_89e924c44eb640448848f5b5960dca76 [id=f6d0c22dccd041e8b32d343e62c986c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.028072 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aa9ad304134509870dd5b276a66fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.032608 1184 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.032737 1184 ts_tablet_manager.cc:1397] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.006s
I20250110 02:51:24.033344 1184 raft_consensus.cc:357] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.033511 1184 raft_consensus.cc:383] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.033550 1184 raft_consensus.cc:738] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.033720 1184 consensus_queue.cc:260] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.034986 1184 ts_tablet_manager.cc:1428] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:24.034947 891 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a619e16df4361a5cb9c077d8f940d (DEFAULT_TABLE table=default.Table_c7ce5ef97f694508890c6699a639c198 [id=a0731c37bdfd4becacd651de1d0bed96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.035280 1184 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.035331 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9a619e16df4361a5cb9c077d8f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.036612 1184 tablet_bootstrap.cc:654] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.041942 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:24.042222 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:24.042780 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:24.043054 1188 tablet_bootstrap.cc:492] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.043175 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]). Expected version 4 got 0
I20250110 02:51:24.043208 1188 ts_tablet_manager.cc:1397] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.005s
I20250110 02:51:24.043550 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:24.043920 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:24.043853 1188 raft_consensus.cc:357] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.044020 1188 raft_consensus.cc:383] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.044083 1188 raft_consensus.cc:738] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.044256 1188 consensus_queue.cc:260] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.044612 747 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a619e16df4361a5cb9c077d8f940d (DEFAULT_TABLE table=default.Table_c7ce5ef97f694508890c6699a639c198 [id=a0731c37bdfd4becacd651de1d0bed96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.044761 1188 ts_tablet_manager.cc:1428] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:24.045038 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9a619e16df4361a5cb9c077d8f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.045063 1188 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.045226 2024 consensus_queue.cc:1035] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.044063 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:24.046633 1188 tablet_bootstrap.cc:654] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.047183 1184 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.047317 1184 ts_tablet_manager.cc:1397] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250110 02:51:24.047947 1184 raft_consensus.cc:357] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.048221 1184 raft_consensus.cc:383] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.048281 1184 raft_consensus.cc:738] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.048481 1184 consensus_queue.cc:260] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.049044 1184 ts_tablet_manager.cc:1428] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:24.049264 1184 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:24.049265 994 consensus_peers.cc:487] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.050441 1184 tablet_bootstrap.cc:654] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.051179 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:24.051883 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:24.052351 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ec427cc101bd49f6b9d1743e3eeebbd1 (table default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e]). Expected version 1 got 0
I20250110 02:51:24.053081 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:24.053395 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:24.053754 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:24.056732 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:24.056907 1440 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b6757b7ecc514489a0fa7e1ff18206bd (table default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf]). Expected version 2 got 0
I20250110 02:51:24.057224 1448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09] alter complete (version 4)
I20250110 02:51:24.058817 1031 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89478577a1428e8957879d6d52be28 (DEFAULT_TABLE table=default.Table_119ec4ba01fd4c019b13cd28b0a3b82c [id=e2d675e7ff0244c08b1afe3d26070a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.059294 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89478577a1428e8957879d6d52be28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.070506 2186 raft_consensus.cc:491] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.070750 2186 raft_consensus.cc:513] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.071673 2186 leader_election.cc:290] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.077809 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:24.078236 1456 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 63) took 2476 ms. Trace:
I20250110 02:51:24.078389 1456 rpcz_store.cc:276] 0110 02:51:21.601545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.759650 (+1158105us) service_pool.cc:225] Handling call
0110 02:51:22.760028 (+ 378us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:22.760268 (+ 240us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:22.760416 (+ 148us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:22.760469 (+ 53us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:22.773913 (+ 13444us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:24.078184 (+1304271us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.761218 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.761316 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.761322 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:22.761324 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.761438 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.761451 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.761455 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.761527 (+ 72us) tablet.cc:698] Row locks acquired
0110 02:51:22.761529 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:22.761557 (+ 28us) write_op.cc:270] Start()
0110 02:51:22.761582 (+ 25us) write_op.cc:276] Timestamp: P: 1736477482761526 usec, L: 0
0110 02:51:22.761585 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.761736 (+ 151us) log.cc:844] Serialized 405 byte log entry
0110 02:51:22.765198 (+ 3462us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.765588 (+ 390us) write_op.cc:301] APPLY: starting
0110 02:51:22.765603 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.765641 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.765669 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.765716 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.765746 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.765752 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:22.765815 (+ 63us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.765866 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.765884 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:22.765889 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.765949 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.765953 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16749,"child_traces":[["op",{"apply.queue_time_us":333,"apply.run_cpu_time_us":358,"apply.run_wall_time_us":358,"num_ops":1,"prepare.queue_time_us":309,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":614,"raft.queue_time_us":66,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"replication_time_us":3590}]]}
I20250110 02:51:24.079222 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 32dd1e726f1a41d0894737b3ef3caf8f from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:24.079816 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.082665 709 leader_election.cc:343] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.084609 880 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.084748 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.084930 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.085143 2352 tablet_replica.cc:331] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.085170 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.085410 2352 raft_consensus.cc:2238] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.085417 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.085614 2352 raft_consensus.cc:2267] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.085932 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.086097 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.086230 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.086304 2352 ts_tablet_manager.cc:1905] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.086720 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.086871 898 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.086992 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.087136 884 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.087268 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.087417 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.087554 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.088164 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:24.089781 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
I20250110 02:51:24.091156 2359 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250110 02:51:24.092712 1184 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.092787 2352 ts_tablet_manager.cc:1918] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.092819 1184 ts_tablet_manager.cc:1397] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.002s
I20250110 02:51:24.092864 2352 log.cc:1198] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/46a4745356e3423f9d4892d242da1c48
I20250110 02:51:24.093202 2352 ts_tablet_manager.cc:1939] T 46a4745356e3423f9d4892d242da1c48 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.094414 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.094565 893 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.094707 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.094841 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.095690 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f (DEFAULT_TABLE table=default.Table_c34337ea54d740158f4bfc9c10db9a5e [id=ddbf80bc2e5a4dcd9b07d96ccca0b9fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:24.096072 1457 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fd02e9f8cd0748a0a6a3202fbe390ad1 consensus state has no leader
I20250110 02:51:24.096117 1184 raft_consensus.cc:357] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.096273 1184 raft_consensus.cc:383] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.096314 1184 raft_consensus.cc:738] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.096513 1184 consensus_queue.cc:260] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.097378 1184 ts_tablet_manager.cc:1428] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250110 02:51:24.060910 2024 consensus_queue.cc:1035] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20250110 02:51:24.100024 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 76) took 1832 ms. Trace:
I20250110 02:51:24.100155 1456 rpcz_store.cc:276] 0110 02:51:22.267470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.079134 (+1811664us) service_pool.cc:225] Handling call
0110 02:51:24.099651 (+ 20517us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:24.100836 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.101037 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.102026 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 971f97a3be9b49958b1218a275e126d4 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:24.104211 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.104612 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.104792 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105044 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105131 894 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105396 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105544 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105544 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105764 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.105890 893 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106021 894 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106165 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106165 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106408 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106534 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106652 893 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106659 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.106894 893 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107033 884 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107164 887 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107303 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107539 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107539 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107791 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.107932 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.108057 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.108865 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.108876 890 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.111076 2352 tablet_replica.cc:331] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.111315 2352 raft_consensus.cc:2238] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.111467 2352 raft_consensus.cc:2267] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.111892 2352 ts_tablet_manager.cc:1905] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.114708 1184 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.116303 1184 tablet_bootstrap.cc:654] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.118325 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 00aa9ad304134509870dd5b276a66fd8 (DEFAULT_TABLE table=default.Table_89e924c44eb640448848f5b5960dca76 [id=f6d0c22dccd041e8b32d343e62c986c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.122465 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 46a4745356e3423f9d4892d242da1c48 (table default.Table_123de8ace5ed47f7850040c5de4fcbc7 [id=3aad422d584f40df9d9241586adbc66f]) successfully deleted
I20250110 02:51:24.126282 2272 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250110 02:51:24.128507 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de9cf980d9c4117b4ec3ca937cfcf01 already in progress: creating tablet
W20250110 02:51:24.126902 709 consensus_peers.cc:487] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.131680 2352 ts_tablet_manager.cc:1918] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.131765 2352 log.cc:1198] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:24.135838 1036 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a619e16df4361a5cb9c077d8f940d (DEFAULT_TABLE table=default.Table_c7ce5ef97f694508890c6699a639c198 [id=a0731c37bdfd4becacd651de1d0bed96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.136035 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.136252 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.139727 2352 ts_tablet_manager.cc:1939] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.140861 2352 tablet_replica.cc:331] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.141041 2352 raft_consensus.cc:2238] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.141163 2352 raft_consensus.cc:2267] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.141635 2352 ts_tablet_manager.cc:1905] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.147735 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.147913 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148046 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148125 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148325 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148324 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148520 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148664 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.148717 2357 tablet_replica.cc:331] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.148906 2357 raft_consensus.cc:2238] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.149034 755 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.149047 2357 raft_consensus.cc:2267] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.149609 2357 ts_tablet_manager.cc:1905] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.149636 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.149693 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.150095 751 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.150328 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
I20250110 02:51:24.151041 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 971f97a3be9b49958b1218a275e126d4
W20250110 02:51:24.151172 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9289de0630804f9aa60fdaf5f3f15f68 already in progress: creating tablet
W20250110 02:51:24.151015 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 116) took 2766 ms (client timeout 1000 ms). Trace:
W20250110 02:51:24.150997 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:24.152755 1456 catalog_manager.cc:6596] ReplaceTablet: tablet 971f97a3be9b49958b1218a275e126d4 deleted and replaced by tablet 3bff349588d44a7dae691bc1d2e2886a
I20250110 02:51:24.152877 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 77) took 1783 ms. Trace:
I20250110 02:51:24.153144 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.153589 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.155094 1456 rpcz_store.cc:276] 0110 02:51:22.369629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.100416 (+1730787us) service_pool.cc:225] Handling call
0110 02:51:24.152830 (+ 52414us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.119138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.119231 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.119237 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:24.119240 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:24.119375 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.119387 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.119392 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.119481 (+ 89us) tablet.cc:698] Row locks acquired
0110 02:51:24.119484 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:24.119518 (+ 34us) write_op.cc:270] Start()
0110 02:51:24.119550 (+ 32us) write_op.cc:276] Timestamp: P: 1736477484119451 usec, L: 0
0110 02:51:24.119553 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.119740 (+ 187us) log.cc:844] Serialized 622 byte log entry
0110 02:51:24.124937 (+ 5197us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.125044 (+ 107us) write_op.cc:301] APPLY: starting
0110 02:51:24.125057 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.125098 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.128738 (+ 3640us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.128833 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.128897 (+ 64us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:24.128905 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:24.128966 (+ 61us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.129025 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.129042 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:24.129065 (+ 23us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.129141 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.129145 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":488,"apply.run_wall_time_us":4086,"num_ops":2,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":701,"prepare.run_wall_time_us":701,"raft.queue_time_us":1493,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":193,"replication_time_us":5360}]]}
I20250110 02:51:24.156683 2240 consensus_queue.cc:1035] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.160158 2055 consensus_queue.cc:1035] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.160084 2357 ts_tablet_manager.cc:1918] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.160915 2357 log.cc:1198] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:24.161698 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.161975 890 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.162189 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.162371 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.162613 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.162843 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.163087 890 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.163411 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.163638 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.163846 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.164052 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:24.164122 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:24.164294 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.164736 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.165094 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.165285 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.165618 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.165835 890 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.166054 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166180 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166244 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.166229 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166415 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166599 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.166595 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166761 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.166878 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.167028 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.167222 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.168154 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f (DEFAULT_TABLE table=default.Table_c34337ea54d740158f4bfc9c10db9a5e [id=ddbf80bc2e5a4dcd9b07d96ccca0b9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.169320 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.169540 2357 ts_tablet_manager.cc:1939] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.169656 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.170002 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.170451 2357 tablet_replica.cc:331] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.170652 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.170675 2357 raft_consensus.cc:2238] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.173031 2357 raft_consensus.cc:2267] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.173632 2357 ts_tablet_manager.cc:1905] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.174515 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.174986 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.175134 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.176350 2224 ts_tablet_manager.cc:1918] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.176467 2224 log.cc:1198] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:24.178530 752 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.178741 996 consensus_peers.cc:487] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.186549 1184 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.186738 1184 ts_tablet_manager.cc:1397] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.005s
I20250110 02:51:24.187500 1184 raft_consensus.cc:357] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.187812 1184 raft_consensus.cc:383] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.187911 1184 raft_consensus.cc:738] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.188161 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (table default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]) successfully deleted
I20250110 02:51:24.188351 1184 consensus_queue.cc:260] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.189165 2352 ts_tablet_manager.cc:1918] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.189540 1184 ts_tablet_manager.cc:1428] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:24.189846 1184 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.190990 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (table default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]) successfully deleted
I20250110 02:51:24.191234 2352 log.cc:1198] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/0d5ddbc6b02742609bc2ca84c863a9d0
W20250110 02:51:24.191272 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 940445840db3441598eba57d3ccb8c9c already in progress: creating tablet
I20250110 02:51:24.191720 2352 ts_tablet_manager.cc:1939] T 0d5ddbc6b02742609bc2ca84c863a9d0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:24.191792 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
I20250110 02:51:24.192593 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.192850 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.194444 2224 ts_tablet_manager.cc:1939] T 76833ce515cf4ee9bc8fff0f626af220 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:24.194196 1434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 -> Peer 8bc934cc708045e9858ee8f8e64660cd (127.0.104.61:44449): Couldn't send request to peer 8bc934cc708045e9858ee8f8e64660cd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250110 02:51:24.194933 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.195183 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.195173 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.196463 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 0d5ddbc6b02742609bc2ca84c863a9d0 (table default.Table_309dd7a424574e50a1d844d112efa307 [id=a86903b224b342459a4e2cd4225f45d0]) successfully deleted
W20250110 02:51:24.163362 1457 rpcz_store.cc:269] 0110 02:51:21.384323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.394244 (+ 9921us) service_pool.cc:225] Handling call
0110 02:51:24.150475 (+2756231us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.993209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.993316 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.993323 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:21.993325 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.993429 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.993441 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.993446 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:21.993503 (+ 57us) tablet.cc:698] Row locks acquired
0110 02:51:21.993505 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.993533 (+ 28us) write_op.cc:270] Start()
0110 02:51:21.993563 (+ 30us) write_op.cc:276] Timestamp: P: 1736477481993481 usec, L: 0
0110 02:51:21.993567 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.993752 (+ 185us) log.cc:844] Serialized 527 byte log entry
0110 02:51:21.996720 (+ 2968us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.996895 (+ 175us) write_op.cc:301] APPLY: starting
0110 02:51:21.996914 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.996965 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.997017 (+ 52us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.997095 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.997136 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:21.997146 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:21.997231 (+ 85us) log.cc:844] Serialized 23 byte log entry
0110 02:51:21.997296 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.997319 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:21.997328 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:21.997408 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.997413 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":129419,"reported_tablets":261,"child_traces":[["op",{"apply.queue_time_us":120,"apply.run_cpu_time_us":501,"apply.run_wall_time_us":501,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":657,"raft.queue_time_us":825,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":239,"replication_time_us":3117}]]}
I20250110 02:51:24.197415 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.197680 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.197831 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.197777 891 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.198613 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.198802 750 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.199460 2352 tablet_replica.cc:331] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.199528 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.199857 1188 tablet_bootstrap.cc:492] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.200065 1188 ts_tablet_manager.cc:1397] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.155s user 0.006s sys 0.002s
I20250110 02:51:24.200187 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.200759 1188 raft_consensus.cc:357] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.200948 1188 raft_consensus.cc:383] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.201006 1188 raft_consensus.cc:738] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.201238 1188 consensus_queue.cc:260] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.202361 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.203191 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e0b2ec1d5814144bc58261e6c6e8d35 already in progress: creating tablet
I20250110 02:51:24.203466 1457 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_fdcd90a5deda47968fc8ba979edbf5a2" } new_table_name: "default.Table_42e03c7535b24cf7981f2a93a5d719f3" modify_external_catalogs: true
I20250110 02:51:24.205122 1188 ts_tablet_manager.cc:1428] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:24.205368 1188 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.210072 1188 tablet_bootstrap.cc:654] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.211378 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_11510771ec434825b06bd79a30ce92b8 [id=f4cb430290954cba98bf9c6c2e45cd39] alter complete (version 1)
I20250110 02:51:24.199868 2352 raft_consensus.cc:2238] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.211745 2352 raft_consensus.cc:2267] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.212520 2352 ts_tablet_manager.cc:1905] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.209120 1184 tablet_bootstrap.cc:654] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.211476 2357 ts_tablet_manager.cc:1918] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.213536 741 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.215090 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.213667 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.213744 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.213832 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.214960 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.215435 2357 log.cc:1198] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/940445840db3441598eba57d3ccb8c9c
I20250110 02:51:24.215775 2357 ts_tablet_manager.cc:1939] T 940445840db3441598eba57d3ccb8c9c P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.217378 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 940445840db3441598eba57d3ccb8c9c (table default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]) successfully deleted
I20250110 02:51:24.217913 2357 tablet_replica.cc:331] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.218473 2357 raft_consensus.cc:2238] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.219056 2357 raft_consensus.cc:2267] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.219867 2357 ts_tablet_manager.cc:1905] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.226493 2186 raft_consensus.cc:491] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.226619 2186 raft_consensus.cc:513] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.227028 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.227125 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.227056 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.227259 2186 leader_election.cc:290] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.227289 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.227028 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.228044 1457 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d637597047ab4738a17eca2f5409459a consensus state has no leader
I20250110 02:51:24.228189 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.228312 1457 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 78) took 1778 ms. Trace:
I20250110 02:51:24.228408 1457 rpcz_store.cc:276] 0110 02:51:22.449406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.196712 (+1747306us) service_pool.cc:225] Handling call
0110 02:51:24.203722 (+ 7010us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.204008 (+ 286us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:24.218406 (+ 14398us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:24.218531 (+ 125us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:24.227847 (+ 9316us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:24.228244 (+ 397us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.219286 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.219399 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.219405 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:24.219407 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.219515 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.219527 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.219532 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:24.219596 (+ 64us) tablet.cc:698] Row locks acquired
0110 02:51:24.219599 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:24.219630 (+ 31us) write_op.cc:270] Start()
0110 02:51:24.219656 (+ 26us) write_op.cc:276] Timestamp: P: 1736477484219605 usec, L: 0
0110 02:51:24.219661 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.219823 (+ 162us) log.cc:844] Serialized 405 byte log entry
0110 02:51:24.222197 (+ 2374us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.224297 (+ 2100us) write_op.cc:301] APPLY: starting
0110 02:51:24.224310 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.224349 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.224377 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.224421 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.224448 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:24.224455 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:24.224521 (+ 66us) log.cc:844] Serialized 23 byte log entry
0110 02:51:24.224575 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.224592 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:24.224603 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.224673 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.224677 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2039,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":372,"num_ops":1,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":643,"raft.queue_time_us":144,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":201,"replication_time_us":2502}]]}
I20250110 02:51:24.228822 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.229159 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.229328 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.229619 1457 master_service.cc:946] ReplaceTablet: received request to replace tablet b375a932156d4abb908dbf327689b0f9 from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:24.229966 709 leader_election.cc:343] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.230043 1457 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 79) took 1710 ms. Trace:
I20250110 02:51:24.230154 1457 rpcz_store.cc:276] 0110 02:51:22.520039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.229518 (+1709479us) service_pool.cc:225] Handling call
0110 02:51:24.230002 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:24.230616 1457 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_06ee6a222f9f4105a670d8943bf209bf" } modify_external_catalogs: true
I20250110 02:51:24.230798 1457 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_06ee6a222f9f4105a670d8943bf209bf" } modify_external_catalogs: true
I20250110 02:51:24.234575 1184 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.234730 1184 ts_tablet_manager.cc:1397] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20250110 02:51:24.235517 1184 raft_consensus.cc:357] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.235735 1184 raft_consensus.cc:383] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.235805 1184 raft_consensus.cc:738] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.236011 2186 raft_consensus.cc:491] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.236092 2186 raft_consensus.cc:513] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.236531 2186 leader_election.cc:290] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.236979 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f15078fc22f4b32a48c7190b36e2192" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.237612 709 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.237879 1184 consensus_queue.cc:260] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.238555 1184 ts_tablet_manager.cc:1428] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:24.238893 1184 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.240379 1188 tablet_bootstrap.cc:492] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.240511 1188 ts_tablet_manager.cc:1397] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250110 02:51:24.241690 1188 raft_consensus.cc:357] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.241916 1188 raft_consensus.cc:383] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.242022 1188 raft_consensus.cc:738] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.242251 1188 consensus_queue.cc:260] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.242837 1188 ts_tablet_manager.cc:1428] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:24.243108 1188 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.244673 1188 tablet_bootstrap.cc:654] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.246490 1184 tablet_bootstrap.cc:654] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.247429 2352 ts_tablet_manager.cc:1918] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.249198 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.249496 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.249752 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.249958 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250085 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250301 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250538 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250566 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250762 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250821 755 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250941 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.250967 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.251071 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.251205 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.251356 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.251518 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.251674 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.252182 2352 log.cc:1198] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:24.252363 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.252781 2352 ts_tablet_manager.cc:1939] T d6fd370138ac4039be55642f1b4c5607 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.254443 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.254511 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.254747 751 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.254901 750 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.255028 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.255163 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.255503 744 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.255694 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.255926 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 105) took 2125 ms (client timeout 1000 ms). Trace:
I20250110 02:51:24.255970 745 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:24.256053 1439 rpcz_store.cc:269] 0110 02:51:22.130042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.243464 (+113422us) service_pool.cc:225] Handling call
0110 02:51:24.255886 (+2012422us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.631303 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.631411 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.631417 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:22.631421 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:22.631534 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.631546 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.631552 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:22.631645 (+ 93us) tablet.cc:698] Row locks acquired
0110 02:51:22.631648 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:22.631680 (+ 32us) write_op.cc:270] Start()
0110 02:51:22.631709 (+ 29us) write_op.cc:276] Timestamp: P: 1736477482631658 usec, L: 0
0110 02:51:22.631714 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.631882 (+ 168us) log.cc:844] Serialized 901 byte log entry
0110 02:51:22.635009 (+ 3127us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.635169 (+ 160us) write_op.cc:301] APPLY: starting
0110 02:51:22.635183 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.635224 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.635258 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.635328 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.635359 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:22.635366 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.635430 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:22.635484 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.635500 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:22.635505 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.635567 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.635571 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":111322,"reported_tablets":270,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":392,"apply.run_wall_time_us":391,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":683,"raft.queue_time_us":139,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":198,"replication_time_us":3267}]]}
I20250110 02:51:24.249042 2357 ts_tablet_manager.cc:1918] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.256908 2357 log.cc:1198] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/19e4af23720048fe98b3231ba402d45c
I20250110 02:51:24.257197 2357 ts_tablet_manager.cc:1939] T 19e4af23720048fe98b3231ba402d45c P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:24.258628 1449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 123) took 3691 ms (client timeout 1000 ms). Trace:
W20250110 02:51:24.258755 1449 rpcz_store.cc:269] 0110 02:51:20.566900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:20.601919 (+ 35019us) service_pool.cc:225] Handling call
0110 02:51:24.258593 (+3656674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.494456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.494541 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.494547 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:21.494549 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.494632 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.494639 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.494644 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:21.494688 (+ 44us) tablet.cc:698] Row locks acquired
0110 02:51:21.494690 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:21.494718 (+ 28us) write_op.cc:270] Start()
0110 02:51:21.494743 (+ 25us) write_op.cc:276] Timestamp: P: 1736477481494659 usec, L: 0
0110 02:51:21.494748 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.494904 (+ 156us) log.cc:844] Serialized 356 byte log entry
0110 02:51:21.621571 (+126667us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.624143 (+ 2572us) write_op.cc:301] APPLY: starting
0110 02:51:21.624156 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.624183 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.624254 (+ 71us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.624299 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.624324 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:21.624331 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:21.624380 (+ 49us) log.cc:844] Serialized 23 byte log entry
0110 02:51:21.624577 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.624589 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:21.624594 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:21.624713 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.624717 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24367,"reported_tablets":222,"child_traces":[["op",{"apply.queue_time_us":2532,"apply.run_cpu_time_us":326,"apply.run_wall_time_us":565,"num_ops":1,"prepare.queue_time_us":78340,"prepare.run_cpu_time_us":545,"prepare.run_wall_time_us":852,"raft.queue_time_us":27,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":126786}]]}
I20250110 02:51:24.260409 1449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_658b8237dabd45f1a02c1fb92070f6e0" } modify_external_catalogs: true
I20250110 02:51:24.260614 1449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_658b8237dabd45f1a02c1fb92070f6e0" } modify_external_catalogs: true
I20250110 02:51:24.261444 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.261658 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.262111 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.262341 2357 tablet_replica.cc:331] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.263212 896 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.263513 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.263746 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.263935 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.264103 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.264256 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.264430 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.264580 896 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.264760 2378 raft_consensus.cc:491] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.264894 2378 raft_consensus.cc:513] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.264981 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 19e4af23720048fe98b3231ba402d45c (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
W20250110 02:51:24.265281 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08705d260c0a462b94ba6b0dc5121123 already in progress: creating tablet
I20250110 02:51:24.265690 2378 leader_election.cc:290] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.265756 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d6fd370138ac4039be55642f1b4c5607 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]) successfully deleted
I20250110 02:51:24.265844 1849 raft_consensus.cc:491] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:24.266227 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b382d634db9642df8e00c5ccaa5787a9 already in progress: creating tablet
W20250110 02:51:24.266521 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
W20250110 02:51:24.266744 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d970e2fe72b4727968ca605b3e772cc already in progress: creating tablet
W20250110 02:51:24.266956 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302af84fd8374ffa96a7503b54849536 already in progress: creating tablet
I20250110 02:51:24.269004 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269227 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269389 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269529 892 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269683 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269824 892 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269958 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.269958 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270179 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270229 893 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270190 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270344 881 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270395 889 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270480 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270526 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270597 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8940c201aa104b0b8bc8d866859fe8ab with delete_type TABLET_DATA_DELETED (replaced by tablet b2163c343a694f5ca04ac9f7610d25e3) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270710 898 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270835 891 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.270957 896 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f6797b1076468a93fb0b510c553bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:10 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.271077 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.271661 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.272060 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.272428 880 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.272791 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.273161 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.273494 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.262493 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.262739 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.262974 744 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.263746 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.264111 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.264581 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.266638 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76833ce515cf4ee9bc8fff0f626af220" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.275612 764 raft_consensus.cc:2388] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:24.276082 2357 raft_consensus.cc:2238] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:24.276748 2357 raft_consensus.cc:2267] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.276973 2240 raft_consensus.cc:491] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.277158 2240 raft_consensus.cc:513] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.274538 1184 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.277606 2186 raft_consensus.cc:491] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.277769 2186 raft_consensus.cc:513] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.278333 2186 leader_election.cc:290] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.278383 2357 ts_tablet_manager.cc:1905] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.281940 2186 raft_consensus.cc:491] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.282058 2186 raft_consensus.cc:513] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.282617 2186 leader_election.cc:290] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.283424 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb726d2c9a04963b92a3fb625cf84f3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.283747 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.284162 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.284549 709 leader_election.cc:343] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:24.284942 709 leader_election.cc:343] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.285177 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.285534 2352 tablet_replica.cc:331] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.285693 1184 ts_tablet_manager.cc:1397] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.002s
I20250110 02:51:24.286520 1184 raft_consensus.cc:357] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.286680 1184 raft_consensus.cc:383] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.286722 1184 raft_consensus.cc:738] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.287158 2352 raft_consensus.cc:2238] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.287344 2352 raft_consensus.cc:2267] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.287496 1188 tablet_bootstrap.cc:492] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.287847 1188 ts_tablet_manager.cc:1397] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.004s
I20250110 02:51:24.288265 2352 ts_tablet_manager.cc:1905] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.286953 1184 consensus_queue.cc:260] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.288681 1188 raft_consensus.cc:357] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.289180 1188 raft_consensus.cc:383] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.289237 1188 raft_consensus.cc:738] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.289425 1188 consensus_queue.cc:260] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.292060 1188 ts_tablet_manager.cc:1428] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:24.292721 1188 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:24.293071 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3510e86314144a2ca012aacf3d78eacc already in progress: creating tablet
W20250110 02:51:24.293408 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4af23720048fe98b3231ba402d45c already in progress: creating tablet
W20250110 02:51:24.293617 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65285bce60fd4015b085deb7442eea6e already in progress: creating tablet
W20250110 02:51:24.293846 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
W20250110 02:51:24.294131 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f15078fc22f4b32a48c7190b36e2192 already in progress: creating tablet
W20250110 02:51:24.294360 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:24.271776 1443 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 124) took 2559 ms (client timeout 1000 ms). Trace:
W20250110 02:51:24.294726 1443 rpcz_store.cc:269] 0110 02:51:21.712005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.731214 (+ 19209us) service_pool.cc:225] Handling call
0110 02:51:21.737156 (+ 5942us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538df66b8 00007f952fefb716 00007f952fefb445 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:24.271686 (+2534530us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:22.293755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:22.293868 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:22.293874 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:22.293876 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:22.293975 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:22.293989 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:22.293994 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:22.294054 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:22.294057 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:22.294093 (+ 36us) write_op.cc:270] Start()
0110 02:51:22.294122 (+ 29us) write_op.cc:276] Timestamp: P: 1736477482294062 usec, L: 0
0110 02:51:22.294125 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:22.294279 (+ 154us) log.cc:844] Serialized 356 byte log entry
0110 02:51:22.299721 (+ 5442us) op_driver.cc:464] REPLICATION: finished
0110 02:51:22.299843 (+ 122us) write_op.cc:301] APPLY: starting
0110 02:51:22.299858 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:22.299896 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:22.299923 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:22.299979 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:22.300012 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:22.300019 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:22.300095 (+ 76us) log.cc:844] Serialized 23 byte log entry
0110 02:51:22.300154 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.300170 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:22.300176 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:22.300242 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:22.300246 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23475,"reported_tablets":240,"spinlock_wait_cycles":3738368,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":390,"num_ops":1,"prepare.queue_time_us":9608,"prepare.run_cpu_time_us":623,"prepare.run_wall_time_us":623,"raft.queue_time_us":824,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":199,"replication_time_us":5570}]]}
I20250110 02:51:24.295729 2357 ts_tablet_manager.cc:1918] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.295825 2357 log.cc:1198] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:24.296167 2357 ts_tablet_manager.cc:1939] T d6fd370138ac4039be55642f1b4c5607 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.296471 1443 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_8115dcee1787457aab5b9023e08afe00" } new_table_name: "default.Table_09da00340dfa4ef59164793be294d5df" modify_external_catalogs: true
I20250110 02:51:24.296900 1184 ts_tablet_manager.cc:1428] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250110 02:51:24.297092 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d6fd370138ac4039be55642f1b4c5607 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]) successfully deleted
I20250110 02:51:24.297168 1184 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.289968 1439 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_610281233be84be48b3fd966dbbcdec2" } modify_external_catalogs: true
I20250110 02:51:24.298977 1188 tablet_bootstrap.cc:654] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.299924 2357 tablet_replica.cc:331] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.300077 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.300123 2357 raft_consensus.cc:2238] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.300240 2357 raft_consensus.cc:2267] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.310032 1439 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_610281233be84be48b3fd966dbbcdec2" } modify_external_catalogs: true
W20250110 02:51:24.314320 1450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 125) took 1243 ms (client timeout 1000 ms). Trace:
I20250110 02:51:24.266010 1849 raft_consensus.cc:513] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.315019 1184 tablet_bootstrap.cc:654] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.315594 1849 leader_election.cc:290] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.316720 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.316892 764 raft_consensus.cc:2463] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:24.317180 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.317833 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:24.318181 994 leader_election.cc:343] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.318357 2352 ts_tablet_manager.cc:1918] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.318464 2352 log.cc:1198] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/28340f0f9a2043a199c74f13755eb702
I20250110 02:51:24.318748 2352 ts_tablet_manager.cc:1939] T 28340f0f9a2043a199c74f13755eb702 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.319487 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.319866 2352 tablet_replica.cc:331] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.319864 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.320086 2352 raft_consensus.cc:2238] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.320250 2352 raft_consensus.cc:2267] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.321210 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:24.322036 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9a619e16df4361a5cb9c077d8f940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.323139 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 73) took 1272 ms. Trace:
I20250110 02:51:24.323237 2224 rpcz_store.cc:276] 0110 02:51:23.051054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.051116 (+ 62us) service_pool.cc:225] Handling call
0110 02:51:23.968577 (+917461us) ts_tablet_manager.cc:1145] Deleting tablet 76833ce515cf4ee9bc8fff0f626af220
0110 02:51:23.968583 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.021033 (+ 52450us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.176239 (+155206us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.323118 (+146879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":917290}
I20250110 02:51:24.323683 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 74) took 1248 ms. Trace:
I20250110 02:51:24.323751 2224 rpcz_store.cc:276] 0110 02:51:23.075446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.075504 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:24.323566 (+1248062us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:24.323577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.323669 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1247835}
I20250110 02:51:24.323998 2224 tablet_replica.cc:331] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:24.324183 2224 raft_consensus.cc:2238] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:24.324368 2224 pending_rounds.cc:62] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:24.324412 2224 pending_rounds.cc:69] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611772351737856 op_type: NO_OP noop_request { }
I20250110 02:51:24.324527 2224 raft_consensus.cc:2883] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:24.324609 2224 raft_consensus.cc:2267] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.325825 2224 ts_tablet_manager.cc:1905] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.326736 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00aa9ad304134509870dd5b276a66fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.329545 914 raft_consensus.cc:3054] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.329629 908 raft_consensus.cc:3054] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.332806 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.333016 2352 ts_tablet_manager.cc:1905] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.333822 1990 consensus_queue.cc:1035] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.334276 1990 consensus_queue.cc:1035] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.336040 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.319365 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.339637 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.339917 2240 leader_election.cc:290] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:24.342525 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.342955 994 leader_election.cc:343] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.344202 996 leader_election.cc:304] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:24.346161 2055 raft_consensus.cc:2798] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:24.346232 2055 raft_consensus.cc:491] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:24.346282 2055 raft_consensus.cc:3054] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:24.348167 996 consensus_peers.cc:487] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f229f546a8d4a09b93f8b46c893d14a. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.350615 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.350848 764 raft_consensus.cc:2463] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:24.351459 996 leader_election.cc:304] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:24.352097 1849 raft_consensus.cc:2798] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:24.352167 1849 raft_consensus.cc:491] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:24.352219 1849 raft_consensus.cc:3054] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.354936 2345 consensus_queue.cc:1035] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.348634 2289 consensus_queue.cc:1035] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20250110 02:51:24.364559 709 consensus_peers.cc:487] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.370244 1184 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.370405 1184 ts_tablet_manager.cc:1397] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20250110 02:51:24.372244 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.373183 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.383268 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.383281 1184 raft_consensus.cc:357] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.383486 1184 raft_consensus.cc:383] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.383536 1184 raft_consensus.cc:738] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.384050 2357 ts_tablet_manager.cc:1905] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.389362 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.384682 1184 consensus_queue.cc:260] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.383144 1849 raft_consensus.cc:513] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.390482 1184 ts_tablet_manager.cc:1428] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250110 02:51:24.390729 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.390820 1184 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.393301 2055 raft_consensus.cc:513] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.394642 2055 leader_election.cc:290] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.395553 1188 tablet_bootstrap.cc:492] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.395660 1188 ts_tablet_manager.cc:1397] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.002s
I20250110 02:51:24.396222 1188 raft_consensus.cc:357] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.396370 1188 raft_consensus.cc:383] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.396416 1188 raft_consensus.cc:738] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.396595 1188 consensus_queue.cc:260] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.398277 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:24.399096 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.400161 1188 ts_tablet_manager.cc:1428] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:24.402010 2432 consensus_queue.cc:1035] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.402722 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:24.402894 768 raft_consensus.cc:3054] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:24.402937 994 leader_election.cc:343] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.405238 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.405491 768 raft_consensus.cc:2463] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:24.405751 2186 raft_consensus.cc:491] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.405841 2186 raft_consensus.cc:513] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.406461 2186 leader_election.cc:290] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.406607 1849 leader_election.cc:290] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:24.407063 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.407204 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:24.407720 709 leader_election.cc:343] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.408485 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:24.408653 772 raft_consensus.cc:3054] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:24.408689 994 leader_election.cc:343] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.409819 2352 ts_tablet_manager.cc:1918] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.409907 2352 log.cc:1198] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:24.410215 2352 ts_tablet_manager.cc:1939] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.411046 772 raft_consensus.cc:2463] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:24.421777 1188 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.423514 1188 tablet_bootstrap.cc:654] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.424465 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f (DEFAULT_TABLE table=default.Table_c34337ea54d740158f4bfc9c10db9a5e [id=ddbf80bc2e5a4dcd9b07d96ccca0b9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.439920 1184 tablet_bootstrap.cc:654] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.442500 1055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.442782 1054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.443038 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:24.443974 711 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:24.444087 711 leader_election.cc:304] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.444418 2199 consensus_queue.cc:1035] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:24.444562 711 leader_election.cc:343] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.444634 711 leader_election.cc:304] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:24.444898 853 consensus_peers.cc:487] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.444880 2253 raft_consensus.cc:2743] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:24.445025 853 leader_election.cc:343] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:24.445055 2253 raft_consensus.cc:2743] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.445106 853 leader_election.cc:304] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:24.445235 853 raft_consensus.cc:2681] T 76833ce515cf4ee9bc8fff0f626af220 P 85f9597077934f1a9814d284ed06d13d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 02:51:24.444909 2199 consensus_queue.cc:1035] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 02:51:24.449975 2377 raft_consensus.cc:491] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.450114 2377 raft_consensus.cc:513] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.450815 2377 leader_election.cc:290] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:24.452888 853 leader_election.cc:343] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 940445840db3441598eba57d3ccb8c9c
I20250110 02:51:24.454995 1188 tablet_bootstrap.cc:492] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.455114 1188 ts_tablet_manager.cc:1397] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.001s
I20250110 02:51:24.455868 1188 raft_consensus.cc:357] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.456645 1188 raft_consensus.cc:383] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.456701 2352 tablet_replica.cc:331] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.319260 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 28340f0f9a2043a199c74f13755eb702 (table default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]) successfully deleted
W20250110 02:51:24.376022 1444 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 104) took 3365 ms (client timeout 1000 ms). Trace:
I20250110 02:51:24.405057 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 76833ce515cf4ee9bc8fff0f626af220 (table default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]) successfully deleted
W20250110 02:51:24.457003 1450 rpcz_store.cc:269] 0110 02:51:23.070772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.073090 (+ 2318us) service_pool.cc:225] Handling call
0110 02:51:24.314276 (+1241186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58570,"reported_tablets":251}
W20250110 02:51:24.457177 1433 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 52146.8us
W20250110 02:51:24.457439 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65285bce60fd4015b085deb7442eea6e already in progress: creating tablet
W20250110 02:51:24.457088 1429 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 137835us
W20250110 02:51:24.458976 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 240b64d551a548e884ee2a7238c0363a already in progress: creating tablet
W20250110 02:51:24.457080 1444 rpcz_store.cc:269] 0110 02:51:21.010615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:21.112429 (+101814us) service_pool.cc:225] Handling call
0110 02:51:24.375978 (+3263549us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:21.823091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:21.823175 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:21.823180 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:21.823182 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:21.823322 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:21.823334 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:21.823339 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:21.823432 (+ 93us) tablet.cc:698] Row locks acquired
0110 02:51:21.823435 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:21.823467 (+ 32us) write_op.cc:270] Start()
0110 02:51:21.823498 (+ 31us) write_op.cc:276] Timestamp: P: 1736477481823431 usec, L: 0
0110 02:51:21.823502 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:21.823712 (+ 210us) log.cc:844] Serialized 901 byte log entry
0110 02:51:21.830796 (+ 7084us) op_driver.cc:464] REPLICATION: finished
0110 02:51:21.830954 (+ 158us) write_op.cc:301] APPLY: starting
0110 02:51:21.830969 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:21.831016 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:21.831065 (+ 49us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:21.831148 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:21.831182 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:21.831188 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:21.831260 (+ 72us) log.cc:844] Serialized 29 byte log entry
0110 02:51:21.831317 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.831338 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:21.831345 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:21.831419 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:21.831423 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":239304,"reported_tablets":249,"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":454,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":710,"prepare.run_wall_time_us":709,"raft.queue_time_us":809,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":251,"replication_time_us":7253}]]}
W20250110 02:51:24.459201 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:24.459381 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 76833ce515cf4ee9bc8fff0f626af220 (table default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]) successfully deleted
I20250110 02:51:24.459478 1444 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_c2fd603cae0f4558877bff141025cd02" } modify_external_catalogs: true
I20250110 02:51:24.459705 1444 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_c2fd603cae0f4558877bff141025cd02" } modify_external_catalogs: true
W20250110 02:51:24.460247 1433 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d637597047ab4738a17eca2f5409459a consensus state has no leader
I20250110 02:51:24.472558 2352 raft_consensus.cc:2238] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.472590 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7259f9abf4e74ecbaa7c91c6c8d4147f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.477701 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb726d2c9a04963b92a3fb625cf84f3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.478092 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.478318 2258 tablet.cc:1721] T 3c0d22fd75fb474ebf07b624499a6f7b P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250110 02:51:24.480705 996 leader_election.cc:304] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:24.487195 1450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_35002d6ab8564fe1b93188d7ce973d3f" } new_table_name: "default.Table_74bf258302164b9eaccfd22680070c50" modify_external_catalogs: true
I20250110 02:51:24.483489 2359 tablet.cc:1721] T fa40c9c3f0ef447294b7c1c5f29ba06e P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:24.492589 2352 raft_consensus.cc:2267] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:24.496290 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 51724.6us
W20250110 02:51:24.497213 711 leader_election.cc:343] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.497391 711 leader_election.cc:304] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.497970 2186 raft_consensus.cc:491] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.498078 2186 raft_consensus.cc:513] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.498664 2186 leader_election.cc:290] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.498795 2186 raft_consensus.cc:2743] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.499694 2352 ts_tablet_manager.cc:1905] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.499675 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.500308 709 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:24.500600 711 leader_election.cc:343] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.500684 711 leader_election.cc:304] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.500862 2186 raft_consensus.cc:2743] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:24.501168 711 leader_election.cc:343] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.501255 711 leader_election.cc:304] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.501421 2186 raft_consensus.cc:2743] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:24.501703 711 leader_election.cc:343] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.501792 711 leader_election.cc:304] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.501971 2186 raft_consensus.cc:2743] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.503016 1184 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.503201 1184 ts_tablet_manager.cc:1397] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.000s
I20250110 02:51:24.503192 2258 tablet.cc:1721] T fa40c9c3f0ef447294b7c1c5f29ba06e P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:24.504243 1184 raft_consensus.cc:357] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.504446 1184 raft_consensus.cc:383] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.504523 1184 raft_consensus.cc:738] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.504753 1184 consensus_queue.cc:260] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.505226 1184 ts_tablet_manager.cc:1428] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:24.505506 1184 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.505537 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940445840db3441598eba57d3ccb8c9c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.456705 1188 raft_consensus.cc:738] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:24.506680 1434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:44449: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250110 02:51:24.506794 1188 consensus_queue.cc:260] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.507246 1188 ts_tablet_manager.cc:1428] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.052s user 0.001s sys 0.001s
I20250110 02:51:24.507475 1188 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.509192 1188 tablet_bootstrap.cc:654] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.511833 2352 ts_tablet_manager.cc:1918] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.511984 2352 log.cc:1198] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b375a932156d4abb908dbf327689b0f9
I20250110 02:51:24.512355 2352 ts_tablet_manager.cc:1939] T b375a932156d4abb908dbf327689b0f9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.512856 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b375a932156d4abb908dbf327689b0f9 (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
I20250110 02:51:24.513157 2352 tablet_replica.cc:331] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.513345 2352 raft_consensus.cc:2238] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.513504 2352 raft_consensus.cc:2267] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.515020 2352 ts_tablet_manager.cc:1905] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:24.521513 1450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b9fb775de0154409a0d92990a51d1ee0 consensus state has no leader
I20250110 02:51:24.521669 1450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 85) took 1559 ms. Trace:
I20250110 02:51:24.521771 1450 rpcz_store.cc:276] 0110 02:51:22.961790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.487025 (+1525235us) service_pool.cc:225] Handling call
0110 02:51:24.487496 (+ 471us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.487720 (+ 224us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:24.487907 (+ 187us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:24.487984 (+ 77us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:24.521206 (+ 33222us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.521212 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:24.521285 (+ 73us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:24.521642 (+ 357us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.514513 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.514641 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.514650 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:24.514654 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:24.514768 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.514781 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.514785 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:24.514846 (+ 61us) tablet.cc:698] Row locks acquired
0110 02:51:24.514849 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:24.514881 (+ 32us) write_op.cc:270] Start()
0110 02:51:24.514923 (+ 42us) write_op.cc:276] Timestamp: P: 1736477484514827 usec, L: 0
0110 02:51:24.514926 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.515136 (+ 210us) log.cc:844] Serialized 405 byte log entry
0110 02:51:24.518756 (+ 3620us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.518933 (+ 177us) write_op.cc:301] APPLY: starting
0110 02:51:24.518948 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.518989 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.519019 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.519066 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.519094 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:24.519101 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:24.519167 (+ 66us) log.cc:844] Serialized 23 byte log entry
0110 02:51:24.519219 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.519237 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:24.519242 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":356,"apply.run_wall_time_us":356,"num_ops":1,"prepare.queue_time_us":26052,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":726,"raft.queue_time_us":817,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":178,"replication_time_us":3798}]]}
I20250110 02:51:24.523075 1450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e5f306c95c94407da3a3eff0c31e33a1" } modify_external_catalogs: true
I20250110 02:51:24.523257 1450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e5f306c95c94407da3a3eff0c31e33a1" } modify_external_catalogs: true
I20250110 02:51:24.524919 2357 ts_tablet_manager.cc:1918] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.525048 2357 log.cc:1198] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:24.525336 2357 ts_tablet_manager.cc:1939] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.526345 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (table default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]) successfully deleted
I20250110 02:51:24.527959 2352 ts_tablet_manager.cc:1918] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:24.528452 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 85442.7us
I20250110 02:51:24.528679 2352 log.cc:1198] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:24.530712 2352 ts_tablet_manager.cc:1939] T d20177229f7b4f11875e0007afb1dd00 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.531251 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d20177229f7b4f11875e0007afb1dd00 (table default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]) successfully deleted
I20250110 02:51:24.531582 2352 tablet_replica.cc:331] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.531800 2352 raft_consensus.cc:2238] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.532020 2352 raft_consensus.cc:2267] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.532550 2352 ts_tablet_manager.cc:1905] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.543959 2352 ts_tablet_manager.cc:1918] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.544154 2352 log.cc:1198] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:24.545372 2352 ts_tablet_manager.cc:1939] T 5af3d61366b2432db53b18f2b7bb49d9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.546907 853 leader_election.cc:304] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.547359 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 5af3d61366b2432db53b18f2b7bb49d9 (table default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]) successfully deleted
I20250110 02:51:24.547433 2378 raft_consensus.cc:2743] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.547709 2055 raft_consensus.cc:2798] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:24.548106 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d424b4da66184bd19eccefad80621516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d424b4da66184bd19eccefad80621516 already in progress: creating tablet
W20250110 02:51:24.549827 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a81c031451104d5a93a1672b4b1918c5 already in progress: creating tablet
I20250110 02:51:24.550154 2352 tablet_replica.cc:331] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.550426 2352 raft_consensus.cc:2238] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.557221 2352 raft_consensus.cc:2267] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.557255 2055 raft_consensus.cc:695] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:24.558104 2352 ts_tablet_manager.cc:1905] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.558859 2055 consensus_queue.cc:237] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.560642 2224 ts_tablet_manager.cc:1918] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.566143 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.566519 2224 log.cc:1198] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/240b64d551a548e884ee2a7238c0363a
I20250110 02:51:24.569859 2224 ts_tablet_manager.cc:1939] T 240b64d551a548e884ee2a7238c0363a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:24.570039 2456 raft_consensus.cc:491] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.570545 2456 raft_consensus.cc:513] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.571776 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3510e86314144a2ca012aacf3d78eacc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.571887 2352 ts_tablet_manager.cc:1918] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.571990 2352 log.cc:1198] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8e79175853474cbea951f5a5686bf630
I20250110 02:51:24.572394 2352 ts_tablet_manager.cc:1939] T 8e79175853474cbea951f5a5686bf630 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:24.572391 994 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.572817 2456 leader_election.cc:290] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.573678 2449 raft_consensus.cc:491] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.573817 2449 raft_consensus.cc:513] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.574672 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.575187 709 leader_election.cc:343] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.575608 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8e79175853474cbea951f5a5686bf630 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:24.578351 2352 tablet_replica.cc:331] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.578616 2352 raft_consensus.cc:2238] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.573173 1181 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:24.573544 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 75) took 1497 ms. Trace:
I20250110 02:51:24.579085 2352 raft_consensus.cc:2267] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.579015 2224 rpcz_store.cc:276] 0110 02:51:23.076137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.076398 (+ 261us) service_pool.cc:225] Handling call
0110 02:51:24.323958 (+1247560us) ts_tablet_manager.cc:1145] Deleting tablet 240b64d551a548e884ee2a7238c0363a
0110 02:51:24.323966 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.417798 (+ 93832us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.540684 (+122886us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.573494 (+ 32810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1247379}
I20250110 02:51:24.579576 2224 tablet_replica.cc:331] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:24.579797 2224 raft_consensus.cc:2238] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:24.579852 2352 ts_tablet_manager.cc:1905] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.580138 2224 raft_consensus.cc:2267] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.580385 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 240b64d551a548e884ee2a7238c0363a (table default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]) successfully deleted
I20250110 02:51:24.580740 1181 ts_tablet_manager.cc:1397] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.571s user 0.004s sys 0.004s
I20250110 02:51:24.581897 1181 raft_consensus.cc:357] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.582973 2449 leader_election.cc:290] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.581079 2224 ts_tablet_manager.cc:1905] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.594797 2352 ts_tablet_manager.cc:1918] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.594962 2352 log.cc:1198] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:24.595358 2352 ts_tablet_manager.cc:1939] T d4f1d916a5ea48708c2c1a61c92fe137 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.596382 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d4f1d916a5ea48708c2c1a61c92fe137 (table default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]) successfully deleted
W20250110 02:51:24.597043 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d00698a0a9ee4e5da69a745f4a3aa8b4 already in progress: creating tablet
I20250110 02:51:24.597379 2352 tablet_replica.cc:331] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.597582 2352 raft_consensus.cc:2238] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:24.597934 2352 raft_consensus.cc:2267] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.598559 2224 ts_tablet_manager.cc:1918] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.598737 2352 ts_tablet_manager.cc:1905] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.598783 2224 log.cc:1198] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b375a932156d4abb908dbf327689b0f9
I20250110 02:51:24.599257 2224 ts_tablet_manager.cc:1939] T b375a932156d4abb908dbf327689b0f9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:24.600587 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 76) took 1522 ms. Trace:
I20250110 02:51:24.603150 2224 rpcz_store.cc:276] 0110 02:51:23.077741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.077789 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:24.579502 (+1501713us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:24.579511 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.595428 (+ 15917us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.598461 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.600523 (+ 2062us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1501563}
I20250110 02:51:24.606971 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b375a932156d4abb908dbf327689b0f9 (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
I20250110 02:51:24.602509 1181 raft_consensus.cc:383] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.607288 1181 raft_consensus.cc:738] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.607595 1181 consensus_queue.cc:260] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.608844 2352 ts_tablet_manager.cc:1918] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.608956 2352 log.cc:1198] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:24.609370 2352 ts_tablet_manager.cc:1939] T 3b6d57f3e9a44d178300d3505d504854 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.609823 1181 ts_tablet_manager.cc:1428] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250110 02:51:24.610308 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 3b6d57f3e9a44d178300d3505d504854 (table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]) successfully deleted
I20250110 02:51:24.614321 2352 tablet_replica.cc:331] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.614578 2352 raft_consensus.cc:2238] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.614701 1181 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:24.614836 2352 raft_consensus.cc:2267] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.616729 1181 tablet_bootstrap.cc:654] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.617169 2352 ts_tablet_manager.cc:1905] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.608215 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 77) took 1511 ms. Trace:
W20250110 02:51:24.604923 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 74389.5us
I20250110 02:51:24.613790 1051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.617904 996 leader_election.cc:304] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:24.618146 2024 raft_consensus.cc:2798] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:24.618434 2024 raft_consensus.cc:695] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:24.621595 2224 rpcz_store.cc:276] 0110 02:51:23.097189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.097235 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:24.608055 (+1510820us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:24.608065 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.608155 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510654}
W20250110 02:51:24.622013 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a81c031451104d5a93a1672b4b1918c5 already in progress: creating tablet
W20250110 02:51:24.622478 711 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.619628 2024 consensus_queue.cc:237] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.622602 711 leader_election.cc:304] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.623155 2199 raft_consensus.cc:2743] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:24.624027 711 leader_election.cc:343] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.624303 2359 tablet.cc:1721] T 598bc151d0c14fc6bb28523c535d9366 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:24.624449 711 leader_election.cc:304] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.624634 2199 raft_consensus.cc:2743] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.630545 2272 tablet.cc:1721] T 598bc151d0c14fc6bb28523c535d9366 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:24.632098 896 tablet_service.cc:1467] Processing CreateTablet for tablet 35517bc9ad204530becbdab3a952d3d0 (DEFAULT_TABLE table=default.Table_c091cf6d870b4996ae16024d22e7a450 [id=dfe04d711d9f447690d644cc9fd4a209]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.632447 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35517bc9ad204530becbdab3a952d3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.632756 751 tablet_service.cc:1467] Processing CreateTablet for tablet 35517bc9ad204530becbdab3a952d3d0 (DEFAULT_TABLE table=default.Table_c091cf6d870b4996ae16024d22e7a450 [id=dfe04d711d9f447690d644cc9fd4a209]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.633147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35517bc9ad204530becbdab3a952d3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.622357 2224 tablet_replica.cc:331] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:24.632308 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 35517bc9ad204530becbdab3a952d3d0 (DEFAULT_TABLE table=default.Table_c091cf6d870b4996ae16024d22e7a450 [id=dfe04d711d9f447690d644cc9fd4a209]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:24.640457 994 consensus_peers.cc:487] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.646507 2377 raft_consensus.cc:491] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.646638 2377 raft_consensus.cc:513] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.646802 2357 tablet_replica.cc:331] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.647003 2357 raft_consensus.cc:2238] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.647174 2357 raft_consensus.cc:2267] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.647214 2377 leader_election.cc:290] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.647811 2357 ts_tablet_manager.cc:1905] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.648368 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.649329 2352 ts_tablet_manager.cc:1918] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.649892 2352 log.cc:1198] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:24.650291 2352 ts_tablet_manager.cc:1939] T e3fc9826ef554a5699e365d10d02e97a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.651325 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e3fc9826ef554a5699e365d10d02e97a (table default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]) successfully deleted
I20250110 02:51:24.651643 2352 tablet_replica.cc:331] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.651827 2352 raft_consensus.cc:2238] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:24.652297 2352 raft_consensus.cc:2267] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.652835 2352 ts_tablet_manager.cc:1905] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.655095 1184 tablet_bootstrap.cc:654] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.656911 896 tablet_service.cc:1467] Processing CreateTablet for tablet dc6b2654f86e4686b8a3034de02d11bf (DEFAULT_TABLE table=default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.657721 751 tablet_service.cc:1467] Processing CreateTablet for tablet dc6b2654f86e4686b8a3034de02d11bf (DEFAULT_TABLE table=default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.658586 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6b2654f86e4686b8a3034de02d11bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.658521 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3bff349588d44a7dae691bc1d2e2886a (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.659509 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3bff349588d44a7dae691bc1d2e2886a (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.659821 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff349588d44a7dae691bc1d2e2886a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.668084 891 tablet_service.cc:1467] Processing CreateTablet for tablet 48050e3bd3ee49c29419a41fb119bb74 (DEFAULT_TABLE table=default.Table_cf24cfbc9d274002aa22a9e06810abd3 [id=5649f528f4e0417ab1867cab8aa23a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.669780 749 tablet_service.cc:1467] Processing CreateTablet for tablet 48050e3bd3ee49c29419a41fb119bb74 (DEFAULT_TABLE table=default.Table_cf24cfbc9d274002aa22a9e06810abd3 [id=5649f528f4e0417ab1867cab8aa23a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.670190 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48050e3bd3ee49c29419a41fb119bb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.671358 1184 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.671517 1184 ts_tablet_manager.cc:1397] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.000s
I20250110 02:51:24.672366 1184 raft_consensus.cc:357] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.672608 1184 raft_consensus.cc:383] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.672695 1184 raft_consensus.cc:738] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.672931 1184 consensus_queue.cc:260] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.673622 1184 ts_tablet_manager.cc:1428] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:24.673900 1184 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.678115 1184 tablet_bootstrap.cc:654] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.684502 1849 consensus_queue.cc:1035] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.687642 1184 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.688531 1184 ts_tablet_manager.cc:1397] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
W20250110 02:51:24.688999 996 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:24.689152 996 leader_election.cc:304] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.687881 2357 ts_tablet_manager.cc:1918] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.690093 2357 log.cc:1198] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:24.691365 1188 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.691506 1188 ts_tablet_manager.cc:1397] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.184s user 0.001s sys 0.001s
I20250110 02:51:24.691553 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35517bc9ad204530becbdab3a952d3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.692709 2449 raft_consensus.cc:491] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.692909 2449 raft_consensus.cc:513] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.693612 2449 leader_election.cc:290] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.693931 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.694538 709 leader_election.cc:343] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.696732 2224 raft_consensus.cc:2238] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.700050 2357 ts_tablet_manager.cc:1939] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.700840 2357 tablet_replica.cc:331] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.701061 2357 raft_consensus.cc:2238] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.701231 2357 raft_consensus.cc:2267] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.701226 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]) successfully deleted
I20250110 02:51:24.701299 1024 tablet_service.cc:1467] Processing CreateTablet for tablet dc6b2654f86e4686b8a3034de02d11bf (DEFAULT_TABLE table=default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.701778 2357 ts_tablet_manager.cc:1905] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.703622 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6b2654f86e4686b8a3034de02d11bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.706606 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff349588d44a7dae691bc1d2e2886a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.706831 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6b2654f86e4686b8a3034de02d11bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.707419 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48050e3bd3ee49c29419a41fb119bb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.710588 2456 consensus_queue.cc:1035] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.712230 1184 raft_consensus.cc:357] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.712451 1184 raft_consensus.cc:383] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.712512 1184 raft_consensus.cc:738] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.712735 1184 consensus_queue.cc:260] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.714061 1184 ts_tablet_manager.cc:1428] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250110 02:51:24.714349 1184 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.714869 1188 raft_consensus.cc:357] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.715075 1188 raft_consensus.cc:383] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.715152 1188 raft_consensus.cc:738] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.715343 1188 consensus_queue.cc:260] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.715945 1188 ts_tablet_manager.cc:1428] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250110 02:51:24.716320 1188 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.717860 1188 tablet_bootstrap.cc:654] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.719206 1184 tablet_bootstrap.cc:654] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.723564 2357 ts_tablet_manager.cc:1918] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.723687 2357 log.cc:1198] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:24.723963 2357 ts_tablet_manager.cc:1939] T 9289de0630804f9aa60fdaf5f3f15f68 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.724920 2357 tablet_replica.cc:331] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.725100 2357 raft_consensus.cc:2238] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.725278 2357 raft_consensus.cc:2267] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.725757 2357 ts_tablet_manager.cc:1905] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.714323 2352 ts_tablet_manager.cc:1918] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.725977 2352 log.cc:1198] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:24.726413 2352 ts_tablet_manager.cc:1939] T b43bb790e85343308a1bb1e1110bfb9c P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.727468 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 9289de0630804f9aa60fdaf5f3f15f68 (table default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]) successfully deleted
W20250110 02:51:24.727751 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
I20250110 02:51:24.729902 1849 raft_consensus.cc:2743] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.732611 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 3bff349588d44a7dae691bc1d2e2886a (DEFAULT_TABLE table=default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.733917 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.736604 2345 raft_consensus.cc:491] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.736761 2345 raft_consensus.cc:513] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.737458 2345 leader_election.cc:290] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.739225 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:24.739261 2224 raft_consensus.cc:2267] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.740870 2224 ts_tablet_manager.cc:1905] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.741667 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.742674 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff349588d44a7dae691bc1d2e2886a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.746459 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f7c6ca2cd747e78c24fe0b0bf99bf4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.747308 853 leader_election.cc:304] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.747660 2378 raft_consensus.cc:2743] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.746762 2453 tablet.cc:1721] T 598bc151d0c14fc6bb28523c535d9366 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:24.750985 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 48050e3bd3ee49c29419a41fb119bb74 (DEFAULT_TABLE table=default.Table_cf24cfbc9d274002aa22a9e06810abd3 [id=5649f528f4e0417ab1867cab8aa23a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:24.752897 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48050e3bd3ee49c29419a41fb119bb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:24.756239 1188 tablet_bootstrap.cc:492] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.756366 1188 ts_tablet_manager.cc:1397] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.006s
I20250110 02:51:24.758641 1188 raft_consensus.cc:357] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.758906 1188 raft_consensus.cc:383] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.758956 1188 raft_consensus.cc:738] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.759460 1188 consensus_queue.cc:260] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.766628 1188 ts_tablet_manager.cc:1428] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:24.768378 1849 consensus_queue.cc:1035] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 02:51:24.770612 2378 raft_consensus.cc:491] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.770725 2378 raft_consensus.cc:513] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.771247 2378 leader_election.cc:290] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.773635 1188 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.776597 1188 tablet_bootstrap.cc:654] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.738590 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e4d2bb0b992474d8e5d21abaca5411c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.776775 764 raft_consensus.cc:2388] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:24.782232 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.782713 2224 ts_tablet_manager.cc:1918] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.788586 2199 raft_consensus.cc:491] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.788900 2199 raft_consensus.cc:513] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.789710 2199 leader_election.cc:290] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:24.754525 711 leader_election.cc:343] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.790490 711 leader_election.cc:304] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.791484 2272 tablet.cc:1721] T fa40c9c3f0ef447294b7c1c5f29ba06e P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:24.793399 2199 raft_consensus.cc:2743] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.795449 2224 log.cc:1198] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8e79175853474cbea951f5a5686bf630
I20250110 02:51:24.796005 2224 ts_tablet_manager.cc:1939] T 8e79175853474cbea951f5a5686bf630 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:24.796382 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 78) took 1687 ms. Trace:
I20250110 02:51:24.796533 2224 rpcz_store.cc:276] 0110 02:51:23.108986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.109045 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:24.622269 (+1513224us) ts_tablet_manager.cc:1145] Deleting tablet 8e79175853474cbea951f5a5686bf630
0110 02:51:24.622280 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.745542 (+123262us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.755560 (+ 10018us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:24.796351 (+ 40791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513040}
I20250110 02:51:24.797096 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8e79175853474cbea951f5a5686bf630 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:24.797494 2224 tablet_replica.cc:331] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:24.797852 2224 raft_consensus.cc:2238] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:24.800446 994 consensus_peers.cc:487] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.800659 2224 raft_consensus.cc:2267] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.805263 2352 tablet_replica.cc:331] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.805627 2352 raft_consensus.cc:2238] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.805843 2352 raft_consensus.cc:2267] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.806599 1181 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:24.806613 2352 ts_tablet_manager.cc:1905] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.806774 1181 ts_tablet_manager.cc:1397] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.192s user 0.002s sys 0.002s
I20250110 02:51:24.807447 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.809334 1181 raft_consensus.cc:357] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.809554 1181 raft_consensus.cc:383] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.809631 1181 raft_consensus.cc:738] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.809849 1181 consensus_queue.cc:260] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.811133 1181 ts_tablet_manager.cc:1428] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:24.812096 1181 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:24.813712 1181 tablet_bootstrap.cc:654] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:24.814682 711 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.818202 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b43bb790e85343308a1bb1e1110bfb9c (table default.Table_83b75cdbc7424f579e0297ecd7b0d4a4 [id=27a2d816626544e68abffc59b5980171]) successfully deleted
I20250110 02:51:24.771767 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1390f867a3a8436092d1920daff4c5b0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.818536 2224 ts_tablet_manager.cc:1905] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.773957 2449 raft_consensus.cc:491] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.818854 2449 raft_consensus.cc:513] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.762892 1184 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.819255 1184 ts_tablet_manager.cc:1397] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.105s user 0.006s sys 0.001s
I20250110 02:51:24.821761 853 leader_election.cc:304] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.822106 2378 raft_consensus.cc:2743] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.819444 2449 leader_election.cc:290] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.824616 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:24.825227 711 leader_election.cc:343] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.823755 1184 raft_consensus.cc:357] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.826471 1184 raft_consensus.cc:383] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.826673 2449 raft_consensus.cc:491] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.826795 2449 raft_consensus.cc:513] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.827260 2449 leader_election.cc:290] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.827927 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:24.832033 711 leader_election.cc:343] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.832195 1184 raft_consensus.cc:738] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.832810 1184 consensus_queue.cc:260] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.837141 1184 ts_tablet_manager.cc:1428] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250110 02:51:24.840741 1184 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.845543 1184 tablet_bootstrap.cc:654] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.848618 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:24.848917 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:24.849177 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:24.849495 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
W20250110 02:51:24.849639 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4042c38de4969889f4821176fc7bb already in progress: creating tablet
I20250110 02:51:24.849819 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:24.850006 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:24.850270 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:24.850453 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:24.850703 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:24.850872 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:24.851042 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:24.851313 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:24.851529 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:24.851719 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:24.851846 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:24.852059 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:24.852293 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:24.852782 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:24.852945 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:24.853363 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:24.856591 2289 consensus_queue.cc:1035] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.859985 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.861523 709 leader_election.cc:343] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.861624 709 leader_election.cc:304] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.861788 2449 raft_consensus.cc:2743] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.863482 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.863828 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.864090 709 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.864200 709 leader_election.cc:304] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.864360 2449 raft_consensus.cc:2743] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.864293 2357 ts_tablet_manager.cc:1918] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.864894 2357 log.cc:1198] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:24.865383 2357 ts_tablet_manager.cc:1939] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.866791 2352 ts_tablet_manager.cc:1918] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.866907 2352 log.cc:1198] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:24.867264 2352 ts_tablet_manager.cc:1939] T bcc9a4a842914a6f878d01194cd9a5f7 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.868124 2352 tablet_replica.cc:331] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.868326 2352 raft_consensus.cc:2238] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.868469 2352 raft_consensus.cc:2267] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.869699 1188 tablet_bootstrap.cc:492] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.870271 2352 ts_tablet_manager.cc:1905] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.870296 1181 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:24.870496 1181 ts_tablet_manager.cc:1397] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.003s
I20250110 02:51:24.871271 1181 raft_consensus.cc:357] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.871486 1181 raft_consensus.cc:383] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.871568 1181 raft_consensus.cc:738] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.871698 1188 ts_tablet_manager.cc:1397] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.015s
I20250110 02:51:24.871847 1181 consensus_queue.cc:260] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.872138 2357 tablet_replica.cc:331] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:24.872318 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (table default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]) successfully deleted
I20250110 02:51:24.872989 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet bcc9a4a842914a6f878d01194cd9a5f7 (table default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]) successfully deleted
I20250110 02:51:24.873157 1181 ts_tablet_manager.cc:1428] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:24.874146 1181 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:24.872802 1188 raft_consensus.cc:357] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.874286 1188 raft_consensus.cc:383] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.874346 1188 raft_consensus.cc:738] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.874544 1188 consensus_queue.cc:260] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:24.872704 709 leader_election.cc:343] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.875043 2357 raft_consensus.cc:2238] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.875058 1188 ts_tablet_manager.cc:1428] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:24.875082 709 leader_election.cc:304] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.875169 2357 raft_consensus.cc:2267] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.875252 2449 raft_consensus.cc:2743] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.853497 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:24.878839 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:24.880651 2352 ts_tablet_manager.cc:1918] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.883219 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:24.883472 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:24.883847 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:24.884868 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:24.885392 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:24.885915 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:24.886143 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:24.887432 1184 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.887553 1184 ts_tablet_manager.cc:1397] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20250110 02:51:24.888248 1184 raft_consensus.cc:357] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.888449 1184 raft_consensus.cc:383] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.888509 1184 raft_consensus.cc:738] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.889097 2357 ts_tablet_manager.cc:1905] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.881188 1188 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.897181 1188 tablet_bootstrap.cc:654] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.889292 1184 consensus_queue.cc:260] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.897745 1184 ts_tablet_manager.cc:1428] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:24.898168 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:24.898527 1184 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.900503 2352 log.cc:1198] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:24.900638 1181 tablet_bootstrap.cc:654] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.900816 2352 ts_tablet_manager.cc:1939] T 1390f867a3a8436092d1920daff4c5b0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.901492 1184 tablet_bootstrap.cc:654] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.901618 2352 tablet_replica.cc:331] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.901799 2352 raft_consensus.cc:2238] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.901914 2352 raft_consensus.cc:2267] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.901916 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1390f867a3a8436092d1920daff4c5b0 (table default.Table_f10221708fa5417d8300ada3f8eb0817 [id=81d7d3112de44cae840ee0f1f86e6f2f]) successfully deleted
I20250110 02:51:24.902297 2352 ts_tablet_manager.cc:1905] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.903913 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3c0d22fd75fb474ebf07b624499a6f7b (table default.Table_d6c1d16cc59d472dabdbb7be7378be29 [id=1d00dbb015e142329070adad93cf1bcb]). Expected version 5 got 4
I20250110 02:51:24.906126 2449 raft_consensus.cc:491] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.906229 2449 raft_consensus.cc:513] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.906831 2449 leader_election.cc:290] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.906948 1188 tablet_bootstrap.cc:492] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.907042 1188 ts_tablet_manager.cc:1397] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20250110 02:51:24.907349 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.907655 1188 raft_consensus.cc:357] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.907838 1188 raft_consensus.cc:383] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:24.907831 709 leader_election.cc:343] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.907881 1188 raft_consensus.cc:738] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.908039 1188 consensus_queue.cc:260] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:24.908425 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:24.908931 711 leader_election.cc:343] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.909087 711 leader_election.cc:304] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.909698 2449 raft_consensus.cc:2743] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.910633 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:24.910833 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:24.911276 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:24.911792 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:24.912050 1188 ts_tablet_manager.cc:1428] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:24.912299 1188 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:24.913034 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:24.914146 2378 raft_consensus.cc:491] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.914268 2378 raft_consensus.cc:513] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.914491 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:24.914849 2378 leader_election.cc:290] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.915230 1456 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
W20250110 02:51:24.916141 853 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 240b64d551a548e884ee2a7238c0363a
I20250110 02:51:24.916342 1188 tablet_bootstrap.cc:654] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.916445 2449 raft_consensus.cc:491] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.916558 2449 raft_consensus.cc:513] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.917125 2449 leader_election.cc:290] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.917309 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240b64d551a548e884ee2a7238c0363a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:24.917387 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65285bce60fd4015b085deb7442eea6e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.917510 767 raft_consensus.cc:2388] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:24.917838 2456 raft_consensus.cc:491] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.917850 853 leader_election.cc:304] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:24.918031 709 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.918025 2456 raft_consensus.cc:513] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.918265 2378 raft_consensus.cc:3054] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.918597 2357 ts_tablet_manager.cc:1918] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:24.918836 1439 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet b45b91d084a44c1e8cdf1553967878e2
I20250110 02:51:24.919157 2456 leader_election.cc:290] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:24.919366 1439 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 82) took 2121 ms. Trace:
I20250110 02:51:24.919446 1439 rpcz_store.cc:276] 0110 02:51:22.797517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.289706 (+1492189us) service_pool.cc:225] Handling call
0110 02:51:24.312655 (+ 22949us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.312826 (+ 171us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:24.312917 (+ 91us) catalog_manager.cc:2762] Locking tablets
0110 02:51:24.553234 (+240317us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:24.565735 (+ 12501us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:24.565804 (+ 69us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:24.918790 (+352986us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:24.919315 (+ 525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.555360 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.555531 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.555535 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:24.555537 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.555611 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.555619 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.555623 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.555707 (+ 84us) tablet.cc:698] Row locks acquired
0110 02:51:24.555709 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:24.555731 (+ 22us) write_op.cc:270] Start()
0110 02:51:24.555750 (+ 19us) write_op.cc:276] Timestamp: P: 1736477484555675 usec, L: 0
0110 02:51:24.555753 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.555882 (+ 129us) log.cc:844] Serialized 725 byte log entry
0110 02:51:24.562879 (+ 6997us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.563407 (+ 528us) write_op.cc:301] APPLY: starting
0110 02:51:24.563415 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.563440 (+ 25us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.563464 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.563502 (+ 38us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.563521 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:24.563526 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:24.563564 (+ 38us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.563591 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.563603 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:24.563607 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":495,"apply.run_cpu_time_us":239,"apply.run_wall_time_us":240,"num_ops":2,"prepare.queue_time_us":306,"prepare.run_cpu_time_us":534,"prepare.run_wall_time_us":670,"raft.queue_time_us":36,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":7112,"thread_start_us":47191,"threads_started":1}]]}
I20250110 02:51:24.920663 2378 raft_consensus.cc:2743] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.923699 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.923909 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7397e41f7374e9fb1621dbcf30af703" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:24.924435 994 leader_election.cc:343] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.927192 2352 ts_tablet_manager.cc:1918] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.927299 2352 log.cc:1198] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f6df074974974d81a83420e7a3a78070
I20250110 02:51:24.927639 2352 ts_tablet_manager.cc:1939] T f6df074974974d81a83420e7a3a78070 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.928479 2352 tablet_replica.cc:331] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:24.928591 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f6df074974974d81a83420e7a3a78070 (table default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]) successfully deleted
I20250110 02:51:24.928584 853 leader_election.cc:304] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.929121 2378 raft_consensus.cc:3054] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:24.931015 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:24.931222 996 leader_election.cc:343] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:24.931318 996 leader_election.cc:304] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:24.931660 1181 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:24.931835 1181 ts_tablet_manager.cc:1397] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250110 02:51:24.931833 2378 raft_consensus.cc:2743] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.931885 2289 raft_consensus.cc:2743] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.932140 2289 consensus_queue.cc:1035] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:24.924388 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.932796 1184 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:24.935328 1184 ts_tablet_manager.cc:1397] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250110 02:51:24.936631 1184 raft_consensus.cc:357] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.937001 1184 raft_consensus.cc:383] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.938014 1184 raft_consensus.cc:738] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.939566 1188 tablet_bootstrap.cc:492] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:24.939880 1188 ts_tablet_manager.cc:1397] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20250110 02:51:24.937708 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65285bce60fd4015b085deb7442eea6e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:24.939954 1181 raft_consensus.cc:357] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.940220 1181 raft_consensus.cc:383] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.940289 1181 raft_consensus.cc:738] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:24.940410 711 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.940511 711 leader_election.cc:304] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:24.940510 1181 consensus_queue.cc:260] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.940856 2449 raft_consensus.cc:2743] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:24.928668 2352 raft_consensus.cc:2238] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:24.941103 1181 ts_tablet_manager.cc:1428] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:24.941339 1181 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:24.941797 1188 raft_consensus.cc:357] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.941907 2352 raft_consensus.cc:2267] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:24.942667 2352 ts_tablet_manager.cc:1905] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:24.938483 1184 consensus_queue.cc:260] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.950520 2500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:44449 every 8 connection(s)
I20250110 02:51:24.950671 1184 ts_tablet_manager.cc:1428] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250110 02:51:24.950939 1184 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:24.951875 1188 raft_consensus.cc:383] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:24.952005 1188 raft_consensus.cc:738] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:24.950974 2002 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:44449
I20250110 02:51:24.952323 1184 tablet_bootstrap.cc:654] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:24.954296 2002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/master-1/data/info.pb
I20250110 02:51:24.956336 1449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 1edb4e06e40c43438882e1eb622d788e
I20250110 02:51:24.956882 1449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 81) took 2159 ms. Trace:
I20250110 02:51:24.957156 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskadt4RW/build/asan/bin/kudu as pid 2002
I20250110 02:51:24.956972 1449 rpcz_store.cc:276] 0110 02:51:22.797398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.260250 (+1462852us) service_pool.cc:225] Handling call
0110 02:51:24.260726 (+ 476us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.260879 (+ 153us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:24.260945 (+ 66us) catalog_manager.cc:2762] Locking tablets
0110 02:51:24.545190 (+284245us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:24.550412 (+ 5222us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:24.550494 (+ 82us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:24.956277 (+405783us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:24.956822 (+ 545us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.545924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.546031 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.546036 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:24.546038 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.546122 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.546131 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.546135 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.546222 (+ 87us) tablet.cc:698] Row locks acquired
0110 02:51:24.546224 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:24.546248 (+ 24us) write_op.cc:270] Start()
0110 02:51:24.546270 (+ 22us) write_op.cc:276] Timestamp: P: 1736477484546192 usec, L: 0
0110 02:51:24.546273 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.546422 (+ 149us) log.cc:844] Serialized 756 byte log entry
0110 02:51:24.549306 (+ 2884us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.549440 (+ 134us) write_op.cc:301] APPLY: starting
0110 02:51:24.549453 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.549487 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.549553 (+ 66us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.549607 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.549641 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:24.549650 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:24.549711 (+ 61us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.549766 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.549782 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:24.549787 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.549860 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.549865 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":408,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":584,"raft.queue_time_us":142,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"replication_time_us":3007}]]}
I20250110 02:51:24.957818 2352 ts_tablet_manager.cc:1918] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:24.957914 2352 log.cc:1198] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:24.958231 2352 ts_tablet_manager.cc:1939] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:24.962857 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb4e06e40c43438882e1eb622d788e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.963914 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb4e06e40c43438882e1eb622d788e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.965579 1188 consensus_queue.cc:260] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.965988 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb4e06e40c43438882e1eb622d788e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:24.966094 1188 ts_tablet_manager.cc:1428] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250110 02:51:24.981812 2357 log.cc:1198] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:24.982342 1457 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 24034fb3da6e47978cdc9b02e714e30d
I20250110 02:51:24.983255 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.984254 1444 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 818dcbf83303412d97b097913bcd36c3
I20250110 02:51:24.985172 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:24.986006 2357 ts_tablet_manager.cc:1939] T 53ae5a6572bd4e7e90e92d4b1cf8d820 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:24.987047 1444 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 84) took 2046 ms. Trace:
I20250110 02:51:24.987152 1444 rpcz_store.cc:276] 0110 02:51:22.941249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.459279 (+1518030us) service_pool.cc:225] Handling call
0110 02:51:24.459823 (+ 544us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.459982 (+ 159us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:24.460065 (+ 83us) catalog_manager.cc:2762] Locking tablets
0110 02:51:24.551725 (+ 91660us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:24.563703 (+ 11978us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:24.563777 (+ 74us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:24.984203 (+420426us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:24.986983 (+ 2780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.553465 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.553543 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.553547 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:24.553549 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.553652 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.553661 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.553665 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.553746 (+ 81us) tablet.cc:698] Row locks acquired
0110 02:51:24.553748 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:24.553772 (+ 24us) write_op.cc:270] Start()
0110 02:51:24.553793 (+ 21us) write_op.cc:276] Timestamp: P: 1736477484553715 usec, L: 0
0110 02:51:24.553796 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.553911 (+ 115us) log.cc:844] Serialized 725 byte log entry
0110 02:51:24.562832 (+ 8921us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.563034 (+ 202us) write_op.cc:301] APPLY: starting
0110 02:51:24.563045 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.563075 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.563095 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.563149 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.563173 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:24.563179 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:24.563233 (+ 54us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.563283 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.563297 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:24.563302 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.563353 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.563357 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":12160,"child_traces":[["op",{"apply.queue_time_us":166,"apply.run_cpu_time_us":313,"apply.run_wall_time_us":313,"mutex_wait_us":271,"num_ops":2,"prepare.queue_time_us":1204,"prepare.run_cpu_time_us":512,"prepare.run_wall_time_us":513,"raft.queue_time_us":90,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":9013}]]}
I20250110 02:51:24.987699 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.987932 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:24.988046 1444 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_8f356d58321749be9aed0b015ff38d71" } new_table_name: "default.Table_5a006cecde524c1dadb83743a5782b86" modify_external_catalogs: true
W20250110 02:51:24.990087 709 consensus_peers.cc:487] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e79175853474cbea951f5a5686bf630. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:24.990206 2449 raft_consensus.cc:491] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:24.990289 2449 raft_consensus.cc:513] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:24.990978 2449 leader_election.cc:290] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:24.992137 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:24.992748 1457 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 80) took 2247 ms. Trace:
I20250110 02:51:24.992853 1457 rpcz_store.cc:276] 0110 02:51:22.739370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.230459 (+1491089us) service_pool.cc:225] Handling call
0110 02:51:24.230914 (+ 455us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.231086 (+ 172us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:24.231153 (+ 67us) catalog_manager.cc:2762] Locking tablets
0110 02:51:24.551725 (+320572us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:24.558730 (+ 7005us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:24.558824 (+ 94us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:24.982273 (+423449us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:24.986980 (+ 4707us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.552828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.552920 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.552925 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:24.552927 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.553022 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.553034 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.553039 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.553132 (+ 93us) tablet.cc:698] Row locks acquired
0110 02:51:24.553149 (+ 17us) write_op.cc:260] PREPARE: finished
0110 02:51:24.553182 (+ 33us) write_op.cc:270] Start()
0110 02:51:24.553205 (+ 23us) write_op.cc:276] Timestamp: P: 1736477484553123 usec, L: 0
0110 02:51:24.553208 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.553344 (+ 136us) log.cc:844] Serialized 725 byte log entry
0110 02:51:24.558133 (+ 4789us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.558291 (+ 158us) write_op.cc:301] APPLY: starting
0110 02:51:24.558304 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.558335 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.558393 (+ 58us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.558463 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.558495 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:24.558501 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:24.558564 (+ 63us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.558618 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.558632 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:24.558638 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.558729 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.558734 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":8192,"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":2,"prepare.queue_time_us":551,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":613,"raft.queue_time_us":500,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":167,"replication_time_us":4898}]]}
I20250110 02:51:24.993754 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_83db50070a234025b9d36aa4b1fe952a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:24.994298 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83db50070a234025b9d36aa4b1fe952a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:24.995834 709 leader_election.cc:343] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:24.999537 1188 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.001626 2289 raft_consensus.cc:491] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.001772 2289 raft_consensus.cc:513] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.002355 2289 leader_election.cc:290] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.003301 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.003588 764 raft_consensus.cc:2463] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:25.003644 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.004006 996 leader_election.cc:304] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.005208 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 (table default.Table_564f2d190f904222b9f705210a88ac5c [id=aaad8783039c4a37872bece7c9d947b5]) successfully deleted
W20250110 02:51:25.006606 1441 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 117) took 2507 ms (client timeout 1000 ms). Trace:
W20250110 02:51:25.006729 1441 rpcz_store.cc:269] 0110 02:51:22.499187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:22.528295 (+ 29108us) service_pool.cc:225] Handling call
0110 02:51:25.006567 (+2478272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.396357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.396483 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.396492 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:23.396497 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:23.396692 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.396710 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.396716 (+ 6us) tablet.cc:682] Acquiring locks for 3 operations
0110 02:51:23.396876 (+ 160us) tablet.cc:698] Row locks acquired
0110 02:51:23.396881 (+ 5us) write_op.cc:260] PREPARE: finished
0110 02:51:23.396922 (+ 41us) write_op.cc:270] Start()
0110 02:51:23.396962 (+ 40us) write_op.cc:276] Timestamp: P: 1736477483396871 usec, L: 0
0110 02:51:23.396967 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.397212 (+ 245us) log.cc:844] Serialized 1275 byte log entry
0110 02:51:23.401005 (+ 3793us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.401253 (+ 248us) write_op.cc:301] APPLY: starting
0110 02:51:23.401269 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.401353 (+ 84us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.401390 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.401489 (+ 99us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.401530 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0110 02:51:23.401540 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:23.401625 (+ 85us) log.cc:844] Serialized 35 byte log entry
0110 02:51:23.401687 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.401709 (+ 22us) write_op.cc:454] Released schema lock
0110 02:51:23.401717 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.401823 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.401826 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":299673,"reported_tablets":271,"child_traces":[["op",{"apply.queue_time_us":195,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":536,"num_ops":3,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":978,"prepare.run_wall_time_us":977,"raft.queue_time_us":1046,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":306,"replication_time_us":4008}]]}
I20250110 02:51:25.008004 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_60139b9c4e354551888ab15aa5329832"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.008656 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60139b9c4e354551888ab15aa5329832 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.009406 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.008972 2357 tablet_replica.cc:331] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:25.009907 994 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.010082 2289 raft_consensus.cc:2798] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.010145 2289 raft_consensus.cc:491] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.010203 2289 raft_consensus.cc:3054] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.010955 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.012295 2357 raft_consensus.cc:2238] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.013283 2357 raft_consensus.cc:2267] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.013679 2289 raft_consensus.cc:513] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.015640 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.016453 2359 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:25.016733 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.016908 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.017164 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.017303 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.017834 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.017863 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec1dba54a247d5ba9f9e053aa3b645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.018537 2357 ts_tablet_manager.cc:1905] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.018996 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:25.019594 994 leader_election.cc:343] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.020474 2289 leader_election.cc:290] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.021077 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:25.021277 764 raft_consensus.cc:3054] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.023655 764 raft_consensus.cc:2463] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:25.023823 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.024061 2453 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:25.024518 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.026113 1188 tablet_bootstrap.cc:654] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.026890 2345 raft_consensus.cc:491] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.027009 2345 raft_consensus.cc:513] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.027532 2345 leader_election.cc:290] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.030237 1443 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 83) took 2095 ms. Trace:
I20250110 02:51:25.030397 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
I20250110 02:51:25.025090 1184 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.028090 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.030920 1184 ts_tablet_manager.cc:1397] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20250110 02:51:25.030943 767 raft_consensus.cc:2388] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:25.031075 1450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 86) took 2024 ms. Trace:
I20250110 02:51:25.031163 1450 rpcz_store.cc:276] 0110 02:51:23.006341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.522744 (+1516403us) service_pool.cc:225] Handling call
0110 02:51:24.523360 (+ 616us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.523480 (+ 120us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:24.523534 (+ 54us) catalog_manager.cc:2762] Locking tablets
0110 02:51:25.005024 (+481490us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:25.027884 (+ 22860us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:25.030265 (+ 2381us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:25.030318 (+ 53us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:25.031013 (+ 695us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.023020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.023101 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.023106 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.023109 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.023190 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.023198 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.023202 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.023276 (+ 74us) tablet.cc:698] Row locks acquired
0110 02:51:25.023279 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.023302 (+ 23us) write_op.cc:270] Start()
0110 02:51:25.023326 (+ 24us) write_op.cc:276] Timestamp: P: 1736477485023249 usec, L: 0
0110 02:51:25.023329 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.023459 (+ 130us) log.cc:844] Serialized 725 byte log entry
0110 02:51:25.026929 (+ 3470us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.027481 (+ 552us) write_op.cc:301] APPLY: starting
0110 02:51:25.027492 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.027523 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.027546 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.027588 (+ 42us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.027608 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:25.027614 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.027665 (+ 51us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.027698 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.027735 (+ 37us) write_op.cc:454] Released schema lock
0110 02:51:25.027740 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":527,"apply.run_cpu_time_us":308,"apply.run_wall_time_us":307,"mutex_wait_us":16754,"num_ops":2,"prepare.queue_time_us":17344,"raft.queue_time_us":886,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":3587,"threads_started":1}]]}
I20250110 02:51:25.031606 1184 raft_consensus.cc:357] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.030710 1443 rpcz_store.cc:276] 0110 02:51:22.934767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.296306 (+1361539us) service_pool.cc:225] Handling call
0110 02:51:24.296659 (+ 353us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:24.296876 (+ 217us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:24.306171 (+ 9295us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:24.306267 (+ 96us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:24.312994 (+ 6727us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:25.030179 (+717185us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.306980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.307121 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.307131 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:24.307136 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:24.307277 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.307294 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.307301 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:24.307380 (+ 79us) tablet.cc:698] Row locks acquired
0110 02:51:24.307384 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:24.307422 (+ 38us) write_op.cc:270] Start()
0110 02:51:24.307463 (+ 41us) write_op.cc:276] Timestamp: P: 1736477484307386 usec, L: 0
0110 02:51:24.307469 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.307691 (+ 222us) log.cc:844] Serialized 405 byte log entry
0110 02:51:24.310830 (+ 3139us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.310951 (+ 121us) write_op.cc:301] APPLY: starting
0110 02:51:24.310965 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.311003 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.311026 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.311083 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.311112 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:24.311119 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:24.311197 (+ 78us) log.cc:844] Serialized 23 byte log entry
0110 02:51:24.311249 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.311267 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:24.311275 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.311342 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.311345 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":1,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":860,"raft.queue_time_us":1184,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":259,"replication_time_us":3335}]]}
I20250110 02:51:25.031749 1184 raft_consensus.cc:383] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.031790 1184 raft_consensus.cc:738] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.031975 1184 consensus_queue.cc:260] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.032233 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:25.034420 2272 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:25.036521 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:25.037138 1184 ts_tablet_manager.cc:1428] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:25.037482 1184 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.038749 2357 ts_tablet_manager.cc:1918] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.038857 2357 log.cc:1198] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/240b64d551a548e884ee2a7238c0363a
I20250110 02:51:25.039263 2357 ts_tablet_manager.cc:1939] T 240b64d551a548e884ee2a7238c0363a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.039741 1184 tablet_bootstrap.cc:654] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.040171 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 240b64d551a548e884ee2a7238c0363a (table default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]) successfully deleted
I20250110 02:51:25.040251 2357 tablet_replica.cc:331] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.040422 2357 raft_consensus.cc:2238] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.040539 2357 raft_consensus.cc:2267] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:25.040524 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
I20250110 02:51:25.040948 2357 ts_tablet_manager.cc:1905] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.041728 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
I20250110 02:51:25.042624 2352 tablet_replica.cc:331] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.042857 2352 raft_consensus.cc:2238] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.043004 2352 raft_consensus.cc:2267] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.045919 2357 ts_tablet_manager.cc:1918] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.046013 2357 log.cc:1198] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:25.046262 2357 ts_tablet_manager.cc:1939] T 4d970e2fe72b4727968ca605b3e772cc P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.046721 996 leader_election.cc:304] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:25.047037 2352 ts_tablet_manager.cc:1905] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.047030 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 4d970e2fe72b4727968ca605b3e772cc (table default.Table_5bf91a5e0e004e1880fdf025d5d9ba38 [id=c584ebe680bb449e92795453c9b6ab74]) successfully deleted
I20250110 02:51:25.047191 2289 raft_consensus.cc:2798] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.047287 2357 tablet_replica.cc:331] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.047297 2289 raft_consensus.cc:695] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:25.047719 2357 raft_consensus.cc:2238] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.047868 2357 raft_consensus.cc:2267] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.048295 2357 ts_tablet_manager.cc:1905] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.051136 1184 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.051261 1184 ts_tablet_manager.cc:1397] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250110 02:51:25.047544 2289 consensus_queue.cc:237] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.051928 1184 raft_consensus.cc:357] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.052089 1184 raft_consensus.cc:383] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.052134 1184 raft_consensus.cc:738] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.053089 2449 raft_consensus.cc:491] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.053220 2449 raft_consensus.cc:513] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.053555 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_94cd6ab467f442cba51ee627721ab3e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:25.053779 2449 leader_election.cc:290] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:25.054131 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94cd6ab467f442cba51ee627721ab3e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.054122 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.054600 709 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.056030 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_01904216609c46c581a0b70981481bb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.056459 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01904216609c46c581a0b70981481bb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.067590 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 88) took 2021 ms. Trace:
I20250110 02:51:25.068184 2449 raft_consensus.cc:491] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.067730 1457 rpcz_store.cc:276] 0110 02:51:23.046215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.993506 (+1947291us) service_pool.cc:225] Handling call
0110 02:51:25.030256 (+ 36750us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.049818 (+ 19562us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.067254 (+ 17436us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.067400 (+ 146us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.067506 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.055161 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.055242 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.055248 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:25.055250 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.055331 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.055343 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.055347 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.055417 (+ 70us) tablet.cc:698] Row locks acquired
0110 02:51:25.055419 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.055448 (+ 29us) write_op.cc:270] Start()
0110 02:51:25.055476 (+ 28us) write_op.cc:276] Timestamp: P: 1736477485055389 usec, L: 0
0110 02:51:25.055480 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.055633 (+ 153us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.066299 (+ 10666us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.066525 (+ 226us) write_op.cc:301] APPLY: starting
0110 02:51:25.066546 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.066605 (+ 59us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.066645 (+ 40us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.066789 (+ 144us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.066835 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.066846 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:25.066940 (+ 94us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.067015 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.067048 (+ 33us) write_op.cc:454] Released schema lock
0110 02:51:25.067058 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.067146 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.067152 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":610,"apply.run_wall_time_us":609,"num_ops":2,"prepare.queue_time_us":4769,"prepare.run_cpu_time_us":619,"prepare.run_wall_time_us":1171,"raft.queue_time_us":56,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":230,"replication_time_us":10789}]]}
I20250110 02:51:25.068311 2449 raft_consensus.cc:513] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.068974 2449 leader_election.cc:290] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.068876 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7d36d6648c3f4c45b1672434450cc64b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.069546 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d36d6648c3f4c45b1672434450cc64b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.071061 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.071758 709 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.075906 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.076545 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.078680 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.079066 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.079124 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abbd43638a2c47dcba4f6f90e8676f42" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.079375 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.079649 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:25.081797 711 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.081904 711 leader_election.cc:304] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.082217 853 leader_election.cc:304] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:25.082274 711 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.082633 2378 raft_consensus.cc:3054] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.082394 711 leader_election.cc:304] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.082911 2199 raft_consensus.cc:2743] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.083070 2199 raft_consensus.cc:2743] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:25.083091 711 leader_election.cc:343] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.083184 711 leader_election.cc:304] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.084218 1181 tablet_bootstrap.cc:654] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.084837 2378 raft_consensus.cc:2743] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.086515 2199 raft_consensus.cc:2743] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.091953 2449 raft_consensus.cc:491] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.092093 2449 raft_consensus.cc:513] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.092769 2449 leader_election.cc:290] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.093464 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.093778 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.093999 709 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:25.094203 711 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.094314 711 leader_election.cc:304] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.094652 2449 raft_consensus.cc:2743] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.100062 2378 raft_consensus.cc:491] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.100205 2378 raft_consensus.cc:513] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.100869 2378 leader_election.cc:290] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.101702 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.102109 764 raft_consensus.cc:2463] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.102133 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 90) took 1860 ms. Trace:
I20250110 02:51:25.102550 853 leader_election.cc:304] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.102594 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.102748 1053 raft_consensus.cc:2463] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.102238 1450 rpcz_store.cc:276] 0110 02:51:23.241338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.031654 (+1790316us) service_pool.cc:225] Handling call
0110 02:51:25.053443 (+ 21789us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfbd03 00007f952fefe236 00007f952fefdf65 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:25.054223 (+ 780us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.054916 (+ 693us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.078419 (+ 23503us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.101954 (+ 23535us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.102064 (+ 110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.059964 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.060048 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.060053 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.060055 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.060138 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.060144 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.060148 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.060199 (+ 51us) tablet.cc:698] Row locks acquired
0110 02:51:25.060201 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.060223 (+ 22us) write_op.cc:270] Start()
0110 02:51:25.060242 (+ 19us) write_op.cc:276] Timestamp: P: 1736477485060166 usec, L: 0
0110 02:51:25.060245 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.060352 (+ 107us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.077112 (+ 16760us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.077583 (+ 471us) write_op.cc:301] APPLY: starting
0110 02:51:25.077636 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.077667 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.077696 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.077819 (+ 123us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.077850 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.077859 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:25.077915 (+ 56us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.077960 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.077974 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:25.077980 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":49403776,"child_traces":[["op",{"apply.queue_time_us":435,"apply.run_cpu_time_us":457,"apply.run_wall_time_us":457,"num_ops":2,"prepare.queue_time_us":4658,"prepare.run_cpu_time_us":481,"prepare.run_wall_time_us":1069,"raft.queue_time_us":563,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":16847,"spinlock_wait_cycles":74624,"threads_started":1}]]}
I20250110 02:51:25.102902 2378 raft_consensus.cc:2798] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.103173 2378 raft_consensus.cc:491] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.103317 2378 raft_consensus.cc:3054] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.106197 2378 raft_consensus.cc:513] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.106899 2378 leader_election.cc:290] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:25.108294 994 consensus_peers.cc:487] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.109043 1184 consensus_queue.cc:260] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.109261 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:25.109395 1053 raft_consensus.cc:3054] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.109373 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e758fcd046f4196a3735d4b731544e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:25.109555 1184 ts_tablet_manager.cc:1428] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20250110 02:51:25.109818 1184 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.109567 774 raft_consensus.cc:3054] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.110868 2449 raft_consensus.cc:491] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.110986 2449 raft_consensus.cc:513] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.111549 2449 leader_election.cc:290] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.112078 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51adff7f50c4aec94e1892a50ec36c4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.112176 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51adff7f50c4aec94e1892a50ec36c4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.112531 711 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:25.112634 709 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.112730 709 leader_election.cc:304] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.111578 1184 tablet_bootstrap.cc:654] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.113153 2449 raft_consensus.cc:2743] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.114200 774 raft_consensus.cc:2463] T 6e758fcd046f4196a3735d4b731544e6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.114612 853 leader_election.cc:304] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.114936 2378 raft_consensus.cc:2798] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.115154 2378 raft_consensus.cc:695] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:25.115231 2352 ts_tablet_manager.cc:1918] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.122437 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:25.122606 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.123538 2449 raft_consensus.cc:491] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.123646 2449 raft_consensus.cc:513] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.124203 2449 leader_election.cc:290] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.125519 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.129830 2352 log.cc:1198] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/eff9b10c419242cf85ef1b708bd5a862
W20250110 02:51:25.130584 711 leader_election.cc:343] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.131541 2352 ts_tablet_manager.cc:1939] T eff9b10c419242cf85ef1b708bd5a862 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.132066 2357 ts_tablet_manager.cc:1918] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.132172 2357 log.cc:1198] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/971f97a3be9b49958b1218a275e126d4
I20250110 02:51:25.132453 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 112) took 1025 ms. Trace:
I20250110 02:51:25.134084 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_f38a47fe3eca45f2ab04c52b70cc6c79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.134723 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f38a47fe3eca45f2ab04c52b70cc6c79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.134429 2352 rpcz_store.cc:276] 0110 02:51:24.106935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107135 (+ 200us) service_pool.cc:225] Handling call
0110 02:51:25.042571 (+935436us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:25.042583 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.049809 (+ 7226us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.115137 (+ 65328us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.132432 (+ 17295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":935312}
I20250110 02:51:25.135210 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet eff9b10c419242cf85ef1b708bd5a862 (table default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]) successfully deleted
I20250110 02:51:25.135466 2352 tablet_replica.cc:331] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.135627 2352 raft_consensus.cc:2238] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.135769 2352 raft_consensus.cc:2267] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.115433 2378 consensus_queue.cc:237] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.124855 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.136693 709 leader_election.cc:343] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.136794 709 leader_election.cc:304] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.137039 2449 raft_consensus.cc:2743] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:25.138968 994 consensus_peers.cc:487] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.139587 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 91) took 1895 ms. Trace:
I20250110 02:51:25.139670 1443 rpcz_store.cc:276] 0110 02:51:23.244461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.031849 (+1787388us) service_pool.cc:225] Handling call
0110 02:51:25.055918 (+ 24069us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfbd03 00007f952fefe236 00007f952fefdf65 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:25.056560 (+ 642us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.056967 (+ 407us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.139336 (+ 82369us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.139448 (+ 112us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.139532 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.061156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.061240 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.061247 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.061251 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:25.061336 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.061345 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.061349 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.061416 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:25.061418 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.061443 (+ 25us) write_op.cc:270] Start()
0110 02:51:25.061464 (+ 21us) write_op.cc:276] Timestamp: P: 1736477485061385 usec, L: 0
0110 02:51:25.061467 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.061583 (+ 116us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.132686 (+ 71103us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.136277 (+ 3591us) write_op.cc:301] APPLY: starting
0110 02:51:25.136296 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.136341 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.136397 (+ 56us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.136555 (+ 158us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.136594 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.136602 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:25.136717 (+ 115us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.139184 (+ 2467us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.139248 (+ 64us) write_op.cc:454] Released schema lock
0110 02:51:25.139256 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.139332 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.139336 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":54690048,"child_traces":[["op",{"apply.queue_time_us":3508,"apply.run_cpu_time_us":657,"apply.run_wall_time_us":3053,"num_ops":2,"prepare.queue_time_us":3754,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":652,"raft.queue_time_us":38,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":71189}]]}
I20250110 02:51:25.140220 1184 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.139532 2357 ts_tablet_manager.cc:1939] T 971f97a3be9b49958b1218a275e126d4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.141163 2357 tablet_replica.cc:331] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.141916 2357 raft_consensus.cc:2238] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.142078 2504 sys_catalog.cc:263] Verifying existing consensus state
I20250110 02:51:25.142082 2357 raft_consensus.cc:2267] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.142546 2357 ts_tablet_manager.cc:1905] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.142726 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 971f97a3be9b49958b1218a275e126d4 (table default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]) successfully deleted
I20250110 02:51:25.143231 908 raft_consensus.cc:3054] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.148049 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.149729 1181 tablet_bootstrap.cc:492] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.149869 1181 ts_tablet_manager.cc:1397] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.209s user 0.002s sys 0.002s
I20250110 02:51:25.150527 1181 raft_consensus.cc:357] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.150710 1181 raft_consensus.cc:383] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.150710 1053 raft_consensus.cc:2463] T 6e758fcd046f4196a3735d4b731544e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.150785 1181 raft_consensus.cc:738] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.150978 1181 consensus_queue.cc:260] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.151530 1181 ts_tablet_manager.cc:1428] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:25.153323 2352 ts_tablet_manager.cc:1905] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.156406 1181 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.160475 2357 ts_tablet_manager.cc:1918] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.140324 1184 ts_tablet_manager.cc:1397] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250110 02:51:25.164839 2352 ts_tablet_manager.cc:1918] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.161811 1184 raft_consensus.cc:357] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.170338 1184 raft_consensus.cc:383] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.170456 1184 raft_consensus.cc:738] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.170789 1184 consensus_queue.cc:260] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.171823 1184 ts_tablet_manager.cc:1428] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:25.160564 2357 log.cc:1198] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/65285bce60fd4015b085deb7442eea6e
I20250110 02:51:25.172513 2357 ts_tablet_manager.cc:1939] T 65285bce60fd4015b085deb7442eea6e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.178107 2352 log.cc:1198] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:25.167220 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.180361 2352 ts_tablet_manager.cc:1939] T b2163c343a694f5ca04ac9f7610d25e3 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.182396 2504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Bootstrap starting.
I20250110 02:51:25.169557 1188 tablet_bootstrap.cc:492] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.182987 1188 ts_tablet_manager.cc:1397] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.004s
I20250110 02:51:25.183109 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 115) took 1075 ms. Trace:
I20250110 02:51:25.183223 2352 rpcz_store.cc:276] 0110 02:51:24.107240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107280 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:25.135423 (+1028143us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:25.135433 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.156195 (+ 20762us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.164758 (+ 8563us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.183089 (+ 18331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028035}
I20250110 02:51:25.183794 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b2163c343a694f5ca04ac9f7610d25e3 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:25.187600 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:25.188110 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:25.188560 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:25.189292 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:25.190073 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:25.190670 1181 tablet_bootstrap.cc:654] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.190891 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:25.191295 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:25.196816 2224 ts_tablet_manager.cc:1918] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.196914 2224 log.cc:1198] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:25.197103 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:25.197234 2224 ts_tablet_manager.cc:1939] T d20177229f7b4f11875e0007afb1dd00 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.197500 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:25.197562 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 81) took 2050 ms. Trace:
I20250110 02:51:25.197635 2224 rpcz_store.cc:276] 0110 02:51:23.147172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.147425 (+ 253us) service_pool.cc:225] Handling call
0110 02:51:24.797427 (+1650002us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:24.797438 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:24.822181 (+ 24743us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.196722 (+374541us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.197532 (+ 810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638967}
I20250110 02:51:25.201407 1184 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.198065 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d20177229f7b4f11875e0007afb1dd00 (table default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]) successfully deleted
I20250110 02:51:25.202127 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:25.202888 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:25.203151 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:25.203919 1449 catalog_manager.cc:5526] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:25.204550 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:25.204841 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:25.206535 2224 tablet_replica.cc:331] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.206779 2224 raft_consensus.cc:2238] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.207435 2224 raft_consensus.cc:2267] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.208864 2224 ts_tablet_manager.cc:1905] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.209484 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:25.209148 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 102) took 1103 ms. Trace:
I20250110 02:51:25.210100 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:25.215708 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 79) took 1049 ms. Trace:
I20250110 02:51:25.216114 2357 rpcz_store.cc:276] 0110 02:51:24.165610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166558 (+ 948us) service_pool.cc:225] Handling call
0110 02:51:25.141123 (+974565us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:25.141131 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.157738 (+ 16607us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.160404 (+ 2666us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.215452 (+ 55048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974424}
I20250110 02:51:25.212592 1188 raft_consensus.cc:357] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.217254 1188 raft_consensus.cc:383] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.217312 1188 raft_consensus.cc:738] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.217552 1188 consensus_queue.cc:260] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.218078 1188 ts_tablet_manager.cc:1428] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250110 02:51:25.210770 1181 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.212850 2289 consensus_queue.cc:1035] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:25.218551 1188 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.220017 1181 ts_tablet_manager.cc:1397] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.004s
I20250110 02:51:25.220129 1188 tablet_bootstrap.cc:654] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.220853 1181 raft_consensus.cc:357] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.221815 1181 raft_consensus.cc:383] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.222007 1181 raft_consensus.cc:738] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.222311 1181 consensus_queue.cc:260] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.222888 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.223333 1181 ts_tablet_manager.cc:1428] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:25.223598 1181 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.225064 1181 tablet_bootstrap.cc:654] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.210084 2352 rpcz_store.cc:276] 0110 02:51:24.105880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107512 (+ 1632us) service_pool.cc:225] Handling call
0110 02:51:25.185212 (+1077700us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:25.207390 (+ 22178us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.208946 (+ 1556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075901}
I20250110 02:51:25.225621 2352 tablet_replica.cc:331] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.225867 2352 raft_consensus.cc:2238] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.226050 2352 raft_consensus.cc:2267] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.226701 2352 ts_tablet_manager.cc:1905] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.227497 1184 tablet_bootstrap.cc:654] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.227838 2449 raft_consensus.cc:491] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.227948 2449 raft_consensus.cc:513] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.228528 2449 leader_election.cc:290] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.229422 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.229882 711 leader_election.cc:343] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.229439 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.232311 709 leader_election.cc:343] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.232399 709 leader_election.cc:304] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.237370 2449 raft_consensus.cc:2743] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.238031 2357 tablet_replica.cc:331] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.239048 2357 raft_consensus.cc:2238] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.239579 2357 raft_consensus.cc:2267] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.240172 2224 ts_tablet_manager.cc:1918] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.240310 2224 log.cc:1198] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:25.240659 2224 ts_tablet_manager.cc:1939] T 5af3d61366b2432db53b18f2b7bb49d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.240958 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 82) took 2093 ms. Trace:
I20250110 02:51:25.241128 1188 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.241081 2224 rpcz_store.cc:276] 0110 02:51:23.147240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.147569 (+ 329us) service_pool.cc:225] Handling call
0110 02:51:25.206472 (+2058903us) ts_tablet_manager.cc:1145] Deleting tablet 5af3d61366b2432db53b18f2b7bb49d9
0110 02:51:25.206481 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.220777 (+ 14296us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.240080 (+ 19303us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.240943 (+ 863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047965}
I20250110 02:51:25.241248 1188 ts_tablet_manager.cc:1397] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250110 02:51:25.241277 2357 ts_tablet_manager.cc:1905] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.241959 1188 raft_consensus.cc:357] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.242148 1188 raft_consensus.cc:383] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.242197 1188 raft_consensus.cc:738] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.242748 2224 tablet_replica.cc:331] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.242973 2224 raft_consensus.cc:2238] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.243207 2224 raft_consensus.cc:2267] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.210239 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 89) took 1982 ms. Trace:
I20250110 02:51:25.210242 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 92) took 1950 ms. Trace:
I20250110 02:51:25.246553 1457 rpcz_store.cc:276] 0110 02:51:23.259550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.068606 (+1809056us) service_pool.cc:225] Handling call
0110 02:51:25.101761 (+ 33155us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.102397 (+ 636us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.171364 (+ 68967us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.210076 (+ 38712us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.210185 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.144300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.144420 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.144427 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.144431 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:25.144565 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.144576 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.144583 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.144672 (+ 89us) tablet.cc:698] Row locks acquired
0110 02:51:25.144676 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:25.144711 (+ 35us) write_op.cc:270] Start()
0110 02:51:25.144742 (+ 31us) write_op.cc:276] Timestamp: P: 1736477485144643 usec, L: 0
0110 02:51:25.144747 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.144948 (+ 201us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.170553 (+ 25605us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.170653 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:25.170668 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.170698 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.170720 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.170809 (+ 89us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.170842 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.170854 (+ 12us) write_op.cc:312] APPLY: finished
0110 02:51:25.170905 (+ 51us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.171153 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.171168 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:25.171174 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.171508 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.171513 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":835,"num_ops":2,"prepare.queue_time_us":41298,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":731,"replication_time_us":25777}]]}
I20250110 02:51:25.247469 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_a3b9a4a35b4243988af7de8340b26084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.248085 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3b9a4a35b4243988af7de8340b26084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.248121 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:25.248378 1449 catalog_manager.cc:5526] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:25.248782 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:25.249034 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_09da00340dfa4ef59164793be294d5df [id=0692620395c14593803592e6aade6af3]). Expected version 2 got 1
I20250110 02:51:25.249581 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:25.249884 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:25.250350 2224 ts_tablet_manager.cc:1905] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.210506 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 93) took 1936 ms. Trace:
I20250110 02:51:25.217339 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 65285bce60fd4015b085deb7442eea6e (table default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]) successfully deleted
I20250110 02:51:25.251652 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 5af3d61366b2432db53b18f2b7bb49d9 (table default.Table_ce1eaf1a53cd4620a73549f2734f168b [id=9276639249c642f9b7db0ca43e573820]) successfully deleted
W20250110 02:51:25.225649 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676f5f457d9e4028bfd0b707604c42f5 already in progress: creating tablet
I20250110 02:51:25.252509 1450 rpcz_store.cc:276] 0110 02:51:23.274195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.102963 (+1828768us) service_pool.cc:225] Handling call
0110 02:51:25.133412 (+ 30449us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfbd03 00007f952fefe236 00007f952fefdf65 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:25.134830 (+ 1418us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.135330 (+ 500us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.171390 (+ 36060us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.210352 (+ 38962us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.210454 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.145060 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.145182 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.145189 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.145193 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:25.145287 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.145297 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.145303 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.145372 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:25.145376 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:25.145405 (+ 29us) write_op.cc:270] Start()
0110 02:51:25.145433 (+ 28us) write_op.cc:276] Timestamp: P: 1736477485145337 usec, L: 0
0110 02:51:25.145437 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.145585 (+ 148us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.170613 (+ 25028us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.170748 (+ 135us) write_op.cc:301] APPLY: starting
0110 02:51:25.170762 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.170828 (+ 66us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.170874 (+ 46us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.171093 (+ 219us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.171119 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.171126 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:25.171177 (+ 51us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.171230 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.171247 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:25.171253 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.171314 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.171317 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":69347584,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":560,"apply.run_wall_time_us":560,"num_ops":2,"prepare.queue_time_us":9157,"prepare.run_cpu_time_us":569,"prepare.run_wall_time_us":568,"replication_time_us":25162}]]}
I20250110 02:51:25.254101 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet 7e554d00d05b4fa7a14c9b6159f9d726 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:25.257062 1188 consensus_queue.cc:260] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.257917 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 94) took 1782 ms. Trace:
I20250110 02:51:25.258165 1457 rpcz_store.cc:276] 0110 02:51:23.474404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.247220 (+1772816us) service_pool.cc:225] Handling call
0110 02:51:25.248184 (+ 964us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.248625 (+ 441us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.256921 (+ 8296us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.257027 (+ 106us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.257128 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.249288 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.249376 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.249383 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.249386 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.249471 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.249484 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.249489 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.249573 (+ 84us) tablet.cc:698] Row locks acquired
0110 02:51:25.249576 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.249606 (+ 30us) write_op.cc:270] Start()
0110 02:51:25.249637 (+ 31us) write_op.cc:276] Timestamp: P: 1736477485249581 usec, L: 0
0110 02:51:25.249641 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.249829 (+ 188us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.254853 (+ 5024us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.254973 (+ 120us) write_op.cc:301] APPLY: starting
0110 02:51:25.254987 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.255028 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.255073 (+ 45us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.255185 (+ 112us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.255217 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.255224 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:25.255291 (+ 67us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.255347 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.255366 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:25.255372 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.255432 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.255436 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":2,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":637,"raft.queue_time_us":382,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":209,"replication_time_us":5183}]]}
I20250110 02:51:25.246531 1441 rpcz_store.cc:276] 0110 02:51:23.228183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.007694 (+1779511us) service_pool.cc:225] Handling call
0110 02:51:25.039615 (+ 31921us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.060568 (+ 20953us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.164835 (+104267us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.210011 (+ 45176us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.210185 (+ 174us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.136521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.136615 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.136620 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.136623 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.136708 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.136718 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.136723 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.136822 (+ 99us) tablet.cc:698] Row locks acquired
0110 02:51:25.136825 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.136857 (+ 32us) write_op.cc:270] Start()
0110 02:51:25.136885 (+ 28us) write_op.cc:276] Timestamp: P: 1736477485136790 usec, L: 0
0110 02:51:25.136889 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.137044 (+ 155us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.163693 (+ 26649us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.164250 (+ 557us) write_op.cc:301] APPLY: starting
0110 02:51:25.164259 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.164300 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.164322 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.164407 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.164429 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.164437 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:25.164481 (+ 44us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.164515 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.164526 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:25.164531 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.164584 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.164589 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":528,"apply.run_cpu_time_us":328,"apply.run_wall_time_us":328,"num_ops":2,"prepare.queue_time_us":75589,"prepare.run_cpu_time_us":618,"prepare.run_wall_time_us":7722,"raft.queue_time_us":391,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":26790}]]}
I20250110 02:51:25.258987 2224 ts_tablet_manager.cc:1918] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.259099 2224 log.cc:1198] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:25.259558 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_1929f7bc73fd49cda28f6dc68e8983f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:25.260048 908 raft_consensus.cc:3054] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:25.260135 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1929f7bc73fd49cda28f6dc68e8983f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.261030 1188 ts_tablet_manager.cc:1428] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250110 02:51:25.261286 1188 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.261598 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7cf301ad33854cc0b24ddec01dbdbdd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:25.262194 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cf301ad33854cc0b24ddec01dbdbdd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.267153 1181 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.267316 1181 ts_tablet_manager.cc:1397] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250110 02:51:25.270103 1181 raft_consensus.cc:357] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.270399 1181 raft_consensus.cc:383] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.270505 1181 raft_consensus.cc:738] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.270764 1181 consensus_queue.cc:260] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.271401 1181 ts_tablet_manager.cc:1428] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250110 02:51:25.271893 1181 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.274560 2352 ts_tablet_manager.cc:1918] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.276357 2224 ts_tablet_manager.cc:1939] T 3b6d57f3e9a44d178300d3505d504854 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.276316 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 96) took 1724 ms. Trace:
I20250110 02:51:25.277318 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 83) took 2129 ms. Trace:
I20250110 02:51:25.277638 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 97) took 1491 ms. Trace:
I20250110 02:51:25.276535 1441 rpcz_store.cc:276] 0110 02:51:23.551877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.259366 (+1707489us) service_pool.cc:225] Handling call
0110 02:51:25.260399 (+ 1033us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.260827 (+ 428us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.275740 (+ 14913us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.275746 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:25.275864 (+ 118us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.276056 (+ 192us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.276254 (+ 198us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.265676 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.265761 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.265766 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.265768 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.265894 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.265902 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.265906 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.265970 (+ 64us) tablet.cc:698] Row locks acquired
0110 02:51:25.265974 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:25.266021 (+ 47us) write_op.cc:270] Start()
0110 02:51:25.266055 (+ 34us) write_op.cc:276] Timestamp: P: 1736477485265995 usec, L: 0
0110 02:51:25.266058 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.266192 (+ 134us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.270077 (+ 3885us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.270173 (+ 96us) write_op.cc:301] APPLY: starting
0110 02:51:25.270184 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.270214 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.270234 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.270314 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.270340 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.270346 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.270433 (+ 87us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.270484 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.270501 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:25.270507 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":376,"mutex_wait_us":2219,"num_ops":2,"prepare.queue_time_us":2285,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":579,"replication_time_us":3996}]]}
I20250110 02:51:25.277957 1457 rpcz_store.cc:276] 0110 02:51:23.785680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.261218 (+1475538us) service_pool.cc:225] Handling call
0110 02:51:25.262356 (+ 1138us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.262815 (+ 459us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.276453 (+ 13638us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.276459 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:25.276703 (+ 244us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.276996 (+ 293us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.277501 (+ 505us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.263444 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.263525 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.263531 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:25.263533 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.263618 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.263630 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.263635 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.263704 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:25.263706 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.263735 (+ 29us) write_op.cc:270] Start()
0110 02:51:25.263761 (+ 26us) write_op.cc:276] Timestamp: P: 1736477485263709 usec, L: 0
0110 02:51:25.263765 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.263926 (+ 161us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.267261 (+ 3335us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.267355 (+ 94us) write_op.cc:301] APPLY: starting
0110 02:51:25.267368 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.267407 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.267436 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.267530 (+ 94us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.267576 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.267582 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.267646 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.267699 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.267714 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:25.267720 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":412,"apply.run_wall_time_us":411,"num_ops":2,"prepare.queue_time_us":262,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":559,"raft.queue_time_us":50,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":215,"replication_time_us":3471}]]}
I20250110 02:51:25.279718 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_150713d6bc354b23b78ed7f02cb4da55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:25.281025 1457 master_service.cc:946] ReplaceTablet: received request to replace tablet fe860d60b3194dc5a2e7a94d39d4b8dd from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:25.281859 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150713d6bc354b23b78ed7f02cb4da55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:25.279413 1181 tablet_bootstrap.cc:654] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.284438 2224 rpcz_store.cc:276] 0110 02:51:23.147343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.147392 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:25.242708 (+2095316us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:25.242716 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.256201 (+ 13485us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.258914 (+ 2713us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.277302 (+ 18388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084163}
I20250110 02:51:25.284963 2224 tablet_replica.cc:331] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.285072 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 3b6d57f3e9a44d178300d3505d504854 (table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]) successfully deleted
I20250110 02:51:25.285177 2224 raft_consensus.cc:2238] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.285334 2224 raft_consensus.cc:2267] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.285818 2224 ts_tablet_manager.cc:1905] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.290280 2352 log.cc:1198] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d9018955f3884824beb2e7336987fc1c
I20250110 02:51:25.274976 1188 tablet_bootstrap.cc:654] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.293000 2352 ts_tablet_manager.cc:1939] T d9018955f3884824beb2e7336987fc1c P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.293959 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 116) took 1186 ms. Trace:
I20250110 02:51:25.294284 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:25.295032 2352 rpcz_store.cc:276] 0110 02:51:24.107477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107512 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:25.225580 (+1118068us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:25.225589 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.268566 (+ 42977us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.274478 (+ 5912us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.293941 (+ 19463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117838}
I20250110 02:51:25.295717 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:25.296057 2352 tablet_replica.cc:331] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.296030 2224 ts_tablet_manager.cc:1918] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.296146 2224 log.cc:1198] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:25.296605 2352 raft_consensus.cc:2238] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.299027 2352 raft_consensus.cc:2267] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.299696 2352 ts_tablet_manager.cc:1905] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.300491 2449 raft_consensus.cc:491] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.300604 2449 raft_consensus.cc:513] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.301124 2449 leader_election.cc:290] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.301581 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.302040 709 leader_election.cc:343] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.295970 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d9018955f3884824beb2e7336987fc1c (table default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]) successfully deleted
I20250110 02:51:25.305704 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:25.306203 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 98) took 1497 ms. Trace:
I20250110 02:51:25.307090 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:25.307245 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:25.307569 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:25.308177 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:25.308280 2352 ts_tablet_manager.cc:1918] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.308377 2352 log.cc:1198] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/27a7885a80414f2193400672aa16a911
I20250110 02:51:25.308557 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:25.308719 2352 ts_tablet_manager.cc:1939] T 27a7885a80414f2193400672aa16a911 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.309374 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 113) took 1202 ms. Trace:
I20250110 02:51:25.309449 2352 rpcz_store.cc:276] 0110 02:51:24.107051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107760 (+ 709us) service_pool.cc:225] Handling call
0110 02:51:25.296016 (+1188256us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:25.296024 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.305474 (+ 9450us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.308202 (+ 2728us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.309359 (+ 1157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188147}
I20250110 02:51:25.309860 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 27a7885a80414f2193400672aa16a911 (table default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]) successfully deleted
I20250110 02:51:25.316905 2352 tablet_replica.cc:331] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.317250 1184 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.317379 1184 ts_tablet_manager.cc:1397] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.001s
I20250110 02:51:25.318126 1184 raft_consensus.cc:357] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.318323 1184 raft_consensus.cc:383] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.318415 1184 raft_consensus.cc:738] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.318622 1184 consensus_queue.cc:260] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.307714 1441 rpcz_store.cc:276] 0110 02:51:23.808726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.279006 (+1470280us) service_pool.cc:225] Handling call
0110 02:51:25.282001 (+ 2995us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:25.282726 (+ 725us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:25.304071 (+ 21345us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:25.305064 (+ 993us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:25.305948 (+ 884us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.283940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.284039 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.284053 (+ 14us) write_op.cc:435] Acquired schema lock
0110 02:51:25.284055 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.284138 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.284150 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.284155 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.284226 (+ 71us) tablet.cc:698] Row locks acquired
0110 02:51:25.284229 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.284261 (+ 32us) write_op.cc:270] Start()
0110 02:51:25.284290 (+ 29us) write_op.cc:276] Timestamp: P: 1736477485284232 usec, L: 0
0110 02:51:25.284293 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.284460 (+ 167us) log.cc:844] Serialized 433 byte log entry
0110 02:51:25.288073 (+ 3613us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.288182 (+ 109us) write_op.cc:301] APPLY: starting
0110 02:51:25.288195 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.288247 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.288275 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.288393 (+ 118us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.288425 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:25.288431 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.288494 (+ 63us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.288553 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.288574 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:25.288583 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.288646 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.288650 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":458,"apply.run_wall_time_us":458,"num_ops":2,"prepare.queue_time_us":124,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":613,"raft.queue_time_us":663,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":221,"replication_time_us":3752}]]}
I20250110 02:51:25.318742 2352 raft_consensus.cc:2238] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.319262 2352 raft_consensus.cc:2267] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.320204 2352 ts_tablet_manager.cc:1905] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.327384 1184 ts_tablet_manager.cc:1428] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250110 02:51:25.327479 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:25.322626 1441 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_9ea14d3254a34f3cb6825fc2e407b2d7" } new_table_name: "default.Table_08102d27b4484be78d1304da43bcfd31" modify_external_catalogs: true
I20250110 02:51:25.327692 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:25.327983 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:25.328395 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:25.302714 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.312686 1181 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.328898 1181 ts_tablet_manager.cc:1397] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20250110 02:51:25.328920 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
W20250110 02:51:25.328969 711 leader_election.cc:343] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.329089 711 leader_election.cc:304] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.329471 2449 raft_consensus.cc:2743] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.329694 1184 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.329671 1181 raft_consensus.cc:357] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.329936 1181 raft_consensus.cc:383] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.330026 1181 raft_consensus.cc:738] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.330128 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:25.330281 1181 consensus_queue.cc:260] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.331202 1184 tablet_bootstrap.cc:654] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.331486 1181 ts_tablet_manager.cc:1428] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:25.331840 1181 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.332345 1449 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
W20250110 02:51:25.332970 994 consensus_peers.cc:487] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2163c343a694f5ca04ac9f7610d25e3. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.333851 1181 tablet_bootstrap.cc:654] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.330883 2224 ts_tablet_manager.cc:1939] T b43bb790e85343308a1bb1e1110bfb9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.334822 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 80) took 2187 ms. Trace:
I20250110 02:51:25.334888 2224 rpcz_store.cc:276] 0110 02:51:23.147084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.147764 (+ 680us) service_pool.cc:225] Handling call
0110 02:51:25.284918 (+2137154us) ts_tablet_manager.cc:1145] Deleting tablet b43bb790e85343308a1bb1e1110bfb9c
0110 02:51:25.284928 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.292981 (+ 8053us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.295940 (+ 2959us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.334802 (+ 38862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126341}
I20250110 02:51:25.335346 2224 tablet_replica.cc:331] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.335330 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b43bb790e85343308a1bb1e1110bfb9c (table default.Table_83b75cdbc7424f579e0297ecd7b0d4a4 [id=27a2d816626544e68abffc59b5980171]) successfully deleted
I20250110 02:51:25.335572 2224 raft_consensus.cc:2238] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.335835 2224 raft_consensus.cc:2267] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.336324 2224 ts_tablet_manager.cc:1905] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.340674 2224 ts_tablet_manager.cc:1918] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.340757 2224 log.cc:1198] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:25.343245 2352 ts_tablet_manager.cc:1918] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.343698 2352 log.cc:1198] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:25.344133 2352 ts_tablet_manager.cc:1939] T 019c0d60da5548bb86c93ccd89bf94ff P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.345431 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 107) took 1239 ms. Trace:
I20250110 02:51:25.349007 2357 ts_tablet_manager.cc:1918] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.349159 2357 log.cc:1198] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/302af84fd8374ffa96a7503b54849536
I20250110 02:51:25.349493 2357 ts_tablet_manager.cc:1939] T 302af84fd8374ffa96a7503b54849536 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.349800 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 77) took 1184 ms. Trace:
I20250110 02:51:25.349890 2357 rpcz_store.cc:276] 0110 02:51:24.165405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166727 (+ 1322us) service_pool.cc:225] Handling call
0110 02:51:25.220858 (+1054131us) ts_tablet_manager.cc:1145] Deleting tablet 302af84fd8374ffa96a7503b54849536
0110 02:51:25.237877 (+ 17019us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.323105 (+ 85228us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.348901 (+ 25796us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.349785 (+ 884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050607}
I20250110 02:51:25.350443 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 302af84fd8374ffa96a7503b54849536 (table default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]) successfully deleted
I20250110 02:51:25.359004 1184 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.359328 1184 ts_tablet_manager.cc:1397] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250110 02:51:25.359668 2224 ts_tablet_manager.cc:1939] T d4f1d916a5ea48708c2c1a61c92fe137 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.360337 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 84) took 2212 ms. Trace:
I20250110 02:51:25.360419 2224 rpcz_store.cc:276] 0110 02:51:23.148168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.148215 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:25.335301 (+2187086us) ts_tablet_manager.cc:1145] Deleting tablet d4f1d916a5ea48708c2c1a61c92fe137
0110 02:51:25.335311 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.339010 (+ 3699us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.340622 (+ 1612us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.360320 (+ 19698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176683}
I20250110 02:51:25.360838 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d4f1d916a5ea48708c2c1a61c92fe137 (table default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]) successfully deleted
I20250110 02:51:25.361321 1181 tablet_bootstrap.cc:492] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.361431 1181 ts_tablet_manager.cc:1397] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250110 02:51:25.361768 1184 raft_consensus.cc:357] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.362036 1181 raft_consensus.cc:357] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.362283 1181 raft_consensus.cc:383] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.362346 1181 raft_consensus.cc:738] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.365157 1184 raft_consensus.cc:383] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.365607 1184 raft_consensus.cc:738] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.366968 1184 consensus_queue.cc:260] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.370201 2449 raft_consensus.cc:491] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.370339 2449 raft_consensus.cc:513] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.371069 2449 leader_election.cc:290] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.371678 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.372530 711 leader_election.cc:343] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.373080 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 85) took 2224 ms. Trace:
I20250110 02:51:25.373158 2224 rpcz_store.cc:276] 0110 02:51:23.148698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.148748 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:25.372977 (+2224229us) ts_tablet_manager.cc:1145] Deleting tablet d00698a0a9ee4e5da69a745f4a3aa8b4
0110 02:51:25.372986 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.373065 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2214300}
I20250110 02:51:25.373459 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 86) took 2224 ms. Trace:
I20250110 02:51:25.373525 2224 rpcz_store.cc:276] 0110 02:51:23.149148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.149204 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:25.373378 (+2224174us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:25.373385 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.373448 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2214667}
W20250110 02:51:25.373633 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d00698a0a9ee4e5da69a745f4a3aa8b4 already in progress: creating tablet
I20250110 02:51:25.373754 2224 tablet_replica.cc:331] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.373920 2224 raft_consensus.cc:2238] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.374188 2224 raft_consensus.cc:2267] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:25.374341 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 676f5f457d9e4028bfd0b707604c42f5 already in progress: creating tablet
I20250110 02:51:25.346252 2352 rpcz_store.cc:276] 0110 02:51:24.106440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.107906 (+ 1466us) service_pool.cc:225] Handling call
0110 02:51:25.316839 (+1208933us) ts_tablet_manager.cc:1145] Deleting tablet 019c0d60da5548bb86c93ccd89bf94ff
0110 02:51:25.316850 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.323842 (+ 6992us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.343162 (+ 19320us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.345412 (+ 2250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208828}
I20250110 02:51:25.374678 2352 tablet_replica.cc:331] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.374733 1450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 95) took 1892 ms. Trace:
I20250110 02:51:25.374781 2224 ts_tablet_manager.cc:1905] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.374858 2352 raft_consensus.cc:2238] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.374828 1450 rpcz_store.cc:276] 0110 02:51:23.482548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.253980 (+1771432us) service_pool.cc:225] Handling call
0110 02:51:25.374694 (+120714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:25.375101 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet 8940c201aa104b0b8bc8d866859fe8ab from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:25.375181 2352 raft_consensus.cc:2267] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.375450 1450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 101) took 1006 ms. Trace:
I20250110 02:51:25.375521 1450 rpcz_store.cc:276] 0110 02:51:24.369271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.375038 (+1005767us) service_pool.cc:225] Handling call
0110 02:51:25.375403 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:25.375627 2352 ts_tablet_manager.cc:1905] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.375851 1450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e55025c253874d759eae8dd088e59196" } new_table_name: "default.Table_fb3102e06de047a68de0adb4febe7735" modify_external_catalogs: true
I20250110 02:51:25.376044 2449 raft_consensus.cc:491] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.376161 2449 raft_consensus.cc:513] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.376667 2449 leader_election.cc:290] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.378509 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.379280 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 019c0d60da5548bb86c93ccd89bf94ff (table default.Table_2ca408d476aa49068e9ddf9a597b2e13 [id=6a24a1e09f794a9da8d1dc060ee95162]) successfully deleted
I20250110 02:51:25.372102 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.379453 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.379945 711 leader_election.cc:343] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:25.380177 709 leader_election.cc:343] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.380261 709 leader_election.cc:304] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:25.380611 709 leader_election.cc:343] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.380635 2224 ts_tablet_manager.cc:1918] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.380688 709 leader_election.cc:304] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.380717 2224 log.cc:1198] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:25.381098 2224 ts_tablet_manager.cc:1939] T e3fc9826ef554a5699e365d10d02e97a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.381071 2199 raft_consensus.cc:2743] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.381245 2199 raft_consensus.cc:2743] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.381750 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 87) took 2232 ms. Trace:
I20250110 02:51:25.381829 2224 rpcz_store.cc:276] 0110 02:51:23.149495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.149539 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:25.373720 (+2224181us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:25.373727 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.377824 (+ 4097us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.380567 (+ 2743us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.381733 (+ 1166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19,"tablet-delete.queue_time_us":2215009}
I20250110 02:51:25.382105 2224 tablet_replica.cc:331] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.382242 2224 raft_consensus.cc:2238] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.382359 2224 raft_consensus.cc:2267] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.382939 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e3fc9826ef554a5699e365d10d02e97a (table default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]) successfully deleted
W20250110 02:51:25.383319 996 consensus_peers.cc:487] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 971f97a3be9b49958b1218a275e126d4. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.383589 2352 ts_tablet_manager.cc:1918] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.384172 2352 log.cc:1198] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:25.384507 2224 ts_tablet_manager.cc:1905] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.384528 2352 ts_tablet_manager.cc:1939] T 09f01daf4c67462ba75e1d0cf56cef12 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.384958 2524 raft_consensus.cc:491] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.385063 2524 raft_consensus.cc:513] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.385567 2524 leader_election.cc:290] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.386093 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f15078fc22f4b32a48c7190b36e2192" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.386258 768 raft_consensus.cc:2463] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.386617 853 leader_election.cc:304] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:25.386886 853 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:25.386957 2524 raft_consensus.cc:2798] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.387027 2524 raft_consensus.cc:491] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.387079 2524 raft_consensus.cc:3054] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.388448 1457 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet fe860d60b3194dc5a2e7a94d39d4b8dd
I20250110 02:51:25.389009 1457 catalog_manager.cc:6596] ReplaceTablet: tablet fe860d60b3194dc5a2e7a94d39d4b8dd deleted and replaced by tablet 471862cae972410b9770f7a8742cac0a
I20250110 02:51:25.389122 1457 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 99) took 1460 ms. Trace:
I20250110 02:51:25.389487 2524 raft_consensus.cc:513] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.389223 1457 rpcz_store.cc:276] 0110 02:51:23.928999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.280557 (+1351558us) service_pool.cc:225] Handling call
0110 02:51:25.389093 (+108536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.379972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.380056 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.380064 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:25.380066 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.380170 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.380178 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.380182 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.380247 (+ 65us) tablet.cc:698] Row locks acquired
0110 02:51:25.380250 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.380278 (+ 28us) write_op.cc:270] Start()
0110 02:51:25.380301 (+ 23us) write_op.cc:276] Timestamp: P: 1736477485380240 usec, L: 0
0110 02:51:25.380304 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.380440 (+ 136us) log.cc:844] Serialized 622 byte log entry
0110 02:51:25.387244 (+ 6804us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.387329 (+ 85us) write_op.cc:301] APPLY: starting
0110 02:51:25.387340 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.387369 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.387388 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.387462 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.387522 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.387528 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.387581 (+ 53us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.387626 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.387637 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:25.387642 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.387697 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.387701 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":358,"apply.run_wall_time_us":359,"num_ops":2,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":522,"prepare.run_wall_time_us":522,"replication_time_us":6918}]]}
I20250110 02:51:25.389765 1457 master_service.cc:946] ReplaceTablet: received request to replace tablet 6555b0a4491046bf8f5cd4ee406c3955 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:25.390826 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:25.392760 891 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:25.390880 2524 leader_election.cc:290] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.393747 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 117) took 1286 ms. Trace:
I20250110 02:51:25.393863 2352 rpcz_store.cc:276] 0110 02:51:24.107659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.108028 (+ 369us) service_pool.cc:225] Handling call
0110 02:51:25.374636 (+1266608us) ts_tablet_manager.cc:1145] Deleting tablet 09f01daf4c67462ba75e1d0cf56cef12
0110 02:51:25.374644 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.378332 (+ 3688us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.383513 (+ 5181us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.393731 (+ 10218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266501}
I20250110 02:51:25.394277 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 09f01daf4c67462ba75e1d0cf56cef12 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
I20250110 02:51:25.394310 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f15078fc22f4b32a48c7190b36e2192" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:25.394488 768 raft_consensus.cc:3054] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.394693 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 91) took 1290 ms. Trace:
I20250110 02:51:25.394791 2352 rpcz_store.cc:276] 0110 02:51:24.104375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.108832 (+ 4457us) service_pool.cc:225] Handling call
0110 02:51:25.394578 (+1285746us) ts_tablet_manager.cc:1145] Deleting tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
0110 02:51:25.394588 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.394677 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285565}
W20250110 02:51:25.395255 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 already in progress: creating tablet
I20250110 02:51:25.395581 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 118) took 1286 ms. Trace:
I20250110 02:51:25.395675 2352 rpcz_store.cc:276] 0110 02:51:24.108781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.108846 (+ 65us) service_pool.cc:225] Handling call
0110 02:51:25.395491 (+1286645us) ts_tablet_manager.cc:1145] Deleting tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
0110 02:51:25.395499 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.395567 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286477}
W20250110 02:51:25.396039 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa already in progress: creating tablet
I20250110 02:51:25.396946 768 raft_consensus.cc:2463] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.397307 853 leader_election.cc:304] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.397548 2524 raft_consensus.cc:2798] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:25.398084 853 leader_election.cc:343] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:25.398393 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.399113 2524 raft_consensus.cc:695] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:25.399369 2524 consensus_queue.cc:237] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.403023 2357 tablet_replica.cc:331] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.402706 1184 ts_tablet_manager.cc:1428] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20250110 02:51:25.405071 2224 ts_tablet_manager.cc:1918] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.406442 2224 log.cc:1198] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:25.406838 2224 ts_tablet_manager.cc:1939] T 1390f867a3a8436092d1920daff4c5b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.406881 2352 tablet_replica.cc:331] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.406495 1181 consensus_queue.cc:260] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.407487 2352 raft_consensus.cc:2238] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.408314 1181 ts_tablet_manager.cc:1428] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250110 02:51:25.408720 2352 raft_consensus.cc:2267] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.409325 1181 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.409927 2352 ts_tablet_manager.cc:1905] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.409226 2357 raft_consensus.cc:2238] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.409940 1184 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.411366 1457 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 6555b0a4491046bf8f5cd4ee406c3955
I20250110 02:51:25.412051 1457 catalog_manager.cc:6596] ReplaceTablet: tablet 6555b0a4491046bf8f5cd4ee406c3955 deleted and replaced by tablet b5f162de87044005bdafe93292ff71b8
I20250110 02:51:25.412508 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:25.412576 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:25.414407 1457 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ae4877bba834dbb96ec9675e2aaedd0 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:25.412063 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:25.415149 1188 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.415289 1188 ts_tablet_manager.cc:1397] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.154s user 0.009s sys 0.024s
I20250110 02:51:25.416116 1188 raft_consensus.cc:357] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.416340 1188 raft_consensus.cc:383] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.416409 1188 raft_consensus.cc:738] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:25.417922 994 consensus_peers.cc:487] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9018955f3884824beb2e7336987fc1c. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.416618 1188 consensus_queue.cc:260] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.418818 1188 ts_tablet_manager.cc:1428] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:25.410951 2357 raft_consensus.cc:2267] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.419073 1188 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.419605 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 88) took 2267 ms. Trace:
I20250110 02:51:25.419675 2224 rpcz_store.cc:276] 0110 02:51:23.152170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.152212 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:25.382071 (+2229859us) ts_tablet_manager.cc:1145] Deleting tablet 1390f867a3a8436092d1920daff4c5b0
0110 02:51:25.382079 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.390550 (+ 8471us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.404935 (+ 14385us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.419588 (+ 14653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2207095}
I20250110 02:51:25.419973 2224 tablet_replica.cc:331] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.420125 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1390f867a3a8436092d1920daff4c5b0 (table default.Table_f10221708fa5417d8300ada3f8eb0817 [id=81d7d3112de44cae840ee0f1f86e6f2f]) successfully deleted
I20250110 02:51:25.420176 2224 raft_consensus.cc:2238] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.420313 2224 raft_consensus.cc:2267] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.421653 1188 tablet_bootstrap.cc:654] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.421895 1181 tablet_bootstrap.cc:654] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.422725 2224 ts_tablet_manager.cc:1905] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.424599 1184 tablet_bootstrap.cc:654] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.431686 1188 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.431826 1188 ts_tablet_manager.cc:1397] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20250110 02:51:25.432677 1188 raft_consensus.cc:357] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.433017 1188 raft_consensus.cc:383] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.433086 1188 raft_consensus.cc:738] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.433378 1188 consensus_queue.cc:260] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.433931 1188 ts_tablet_manager.cc:1428] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:25.434173 1188 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.436064 1188 tablet_bootstrap.cc:654] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.437167 2224 ts_tablet_manager.cc:1918] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.437252 2224 log.cc:1198] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:25.437551 2224 ts_tablet_manager.cc:1939] T da8f247c862e4cb68452e230857ee6b0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.438220 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 90) took 2285 ms. Trace:
I20250110 02:51:25.438295 2224 rpcz_store.cc:276] 0110 02:51:23.153063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.153105 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:25.419936 (+2266831us) ts_tablet_manager.cc:1145] Deleting tablet da8f247c862e4cb68452e230857ee6b0
0110 02:51:25.419944 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.425422 (+ 5478us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.437080 (+ 11658us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.438201 (+ 1121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2244893}
I20250110 02:51:25.438810 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet da8f247c862e4cb68452e230857ee6b0 (table default.Table_90e43feb19a343f6b3c9a20876ec8c23 [id=e2a6764b09814b9cb1367abb1a4f5ca9]) successfully deleted
I20250110 02:51:25.439349 1188 tablet_bootstrap.cc:492] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.439456 1188 ts_tablet_manager.cc:1397] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250110 02:51:25.440275 1188 raft_consensus.cc:357] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.440434 1188 raft_consensus.cc:383] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.440485 1188 raft_consensus.cc:738] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.440668 1188 consensus_queue.cc:260] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.441136 1188 ts_tablet_manager.cc:1428] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:25.441350 1188 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.442711 2357 ts_tablet_manager.cc:1905] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.447329 1188 tablet_bootstrap.cc:654] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:25.447798 994 consensus_peers.cc:487] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6df074974974d81a83420e7a3a78070. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.448382 2432 consensus_queue.cc:1035] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:25.456060 2224 tablet_replica.cc:331] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.456315 2224 raft_consensus.cc:2238] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.456645 2224 raft_consensus.cc:2267] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.460013 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.462147 2224 ts_tablet_manager.cc:1905] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.469947 2432 consensus_queue.cc:1035] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:25.474771 2352 ts_tablet_manager.cc:1918] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.474885 2352 log.cc:1198] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:25.475191 2352 ts_tablet_manager.cc:1939] T 7d4422eca5604d149e73c2a669d07af0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.476080 1181 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.476244 1181 ts_tablet_manager.cc:1397] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250110 02:51:25.477034 1181 raft_consensus.cc:357] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.477249 1181 raft_consensus.cc:383] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.477330 1181 raft_consensus.cc:738] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.477548 1181 consensus_queue.cc:260] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.479161 1184 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.479292 1184 ts_tablet_manager.cc:1397] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20250110 02:51:25.480032 1184 raft_consensus.cc:357] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.480294 1184 raft_consensus.cc:383] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.480361 1184 raft_consensus.cc:738] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.480659 1181 ts_tablet_manager.cc:1428] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:25.480647 2449 raft_consensus.cc:491] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.480603 1184 consensus_queue.cc:260] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.480764 2449 raft_consensus.cc:513] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.481249 1184 ts_tablet_manager.cc:1428] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:25.481316 2449 leader_election.cc:290] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.481487 1184 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.481941 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.482101 1061 raft_consensus.cc:2463] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:25.482523 711 leader_election.cc:304] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.483860 2449 raft_consensus.cc:2798] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.483928 2449 raft_consensus.cc:491] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.483979 2449 raft_consensus.cc:3054] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.486464 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.486955 709 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.487489 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 120) took 1325 ms. Trace:
I20250110 02:51:25.486364 2449 raft_consensus.cc:513] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.487586 2352 rpcz_store.cc:276] 0110 02:51:24.161616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.161656 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:25.406820 (+1245164us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:25.406832 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.441028 (+ 34196us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.474672 (+ 33644us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.487442 (+ 12770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235298}
I20250110 02:51:25.488045 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7d4422eca5604d149e73c2a669d07af0 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
I20250110 02:51:25.488394 2352 tablet_replica.cc:331] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.488503 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:25.488596 2352 raft_consensus.cc:2238] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.488960 2352 raft_consensus.cc:2267] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:25.489044 709 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.489665 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:25.489810 1061 raft_consensus.cc:3054] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.491292 2352 ts_tablet_manager.cc:1905] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.492372 1061 raft_consensus.cc:2463] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:25.493084 711 leader_election.cc:304] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:25.493620 2199 raft_consensus.cc:2798] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.493762 2449 leader_election.cc:290] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:25.494009 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd02e9f8cd0748a0a6a3202fbe390ad1 consensus state has no leader
W20250110 02:51:25.496845 994 consensus_peers.cc:487] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.501848 2224 ts_tablet_manager.cc:1918] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.513602 2224 log.cc:1198] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d9018955f3884824beb2e7336987fc1c
I20250110 02:51:25.514043 2224 ts_tablet_manager.cc:1939] T d9018955f3884824beb2e7336987fc1c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.502635 1181 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.508381 2432 raft_consensus.cc:491] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.516096 2432 raft_consensus.cc:513] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.516173 2199 raft_consensus.cc:695] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:25.516433 2199 consensus_queue.cc:237] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.516618 2432 leader_election.cc:290] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.517179 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.517350 764 raft_consensus.cc:2463] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:25.517515 1181 tablet_bootstrap.cc:654] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.518110 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.518246 996 leader_election.cc:304] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.518509 2432 raft_consensus.cc:2798] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.518579 2432 raft_consensus.cc:491] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.518630 2432 raft_consensus.cc:3054] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:25.518723 994 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.521387 2432 raft_consensus.cc:513] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:25.524544 994 consensus_peers.cc:487] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa6057de9da04f2c82b8f25c5b4e2fe4. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.525377 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:25.525524 764 raft_consensus.cc:3054] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.525794 2357 ts_tablet_manager.cc:1918] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.525924 2357 log.cc:1198] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:25.526259 2357 ts_tablet_manager.cc:1939] T b382d634db9642df8e00c5ccaa5787a9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.527153 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 74) took 1362 ms. Trace:
I20250110 02:51:25.527246 2357 rpcz_store.cc:276] 0110 02:51:24.165058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166983 (+ 1925us) service_pool.cc:225] Handling call
0110 02:51:25.402617 (+1235634us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:25.402629 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.496000 (+ 93371us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.525672 (+ 29672us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.527130 (+ 1458us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":791,"tablet-delete.queue_time_us":1235116}
I20250110 02:51:25.527863 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b382d634db9642df8e00c5ccaa5787a9 (table default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]) successfully deleted
I20250110 02:51:25.527977 764 raft_consensus.cc:2463] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:25.528425 2432 leader_election.cc:290] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.528492 1184 tablet_bootstrap.cc:654] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.528394 996 leader_election.cc:304] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.528856 2456 raft_consensus.cc:2798] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.528982 2456 raft_consensus.cc:695] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:25.529204 2456 consensus_queue.cc:237] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.529640 2352 ts_tablet_manager.cc:1918] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.529757 2352 log.cc:1198] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:25.530117 2352 ts_tablet_manager.cc:1939] T da8f247c862e4cb68452e230857ee6b0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.530234 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 83) took 1362 ms. Trace:
I20250110 02:51:25.530354 2357 rpcz_store.cc:276] 0110 02:51:24.167746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.169967 (+ 2221us) service_pool.cc:225] Handling call
0110 02:51:25.529580 (+1359613us) ts_tablet_manager.cc:1145] Deleting tablet c5601ca24b8f428e8a51cbeec3664901
0110 02:51:25.529589 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.530192 (+ 603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359417}
I20250110 02:51:25.530807 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 121) took 1368 ms. Trace:
W20250110 02:51:25.534790 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
I20250110 02:51:25.539466 1181 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.540583 1181 ts_tablet_manager.cc:1397] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20250110 02:51:25.541481 2199 raft_consensus.cc:491] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.541595 2199 raft_consensus.cc:513] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.542121 2199 leader_election.cc:290] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.542497 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.542790 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 89) took 2390 ms. Trace:
I20250110 02:51:25.542865 2224 rpcz_store.cc:276] 0110 02:51:23.152691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.154425 (+ 1734us) service_pool.cc:225] Handling call
0110 02:51:25.456017 (+2301592us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:25.456025 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.476674 (+ 20649us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.501768 (+ 25094us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.542775 (+ 41007us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280926}
W20250110 02:51:25.542980 711 leader_election.cc:343] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.543156 2224 tablet_replica.cc:331] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.543342 2224 raft_consensus.cc:2238] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.543383 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d9018955f3884824beb2e7336987fc1c (table default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]) successfully deleted
I20250110 02:51:25.543485 2224 raft_consensus.cc:2267] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.543957 2224 ts_tablet_manager.cc:1905] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.544200 1184 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.544332 1184 ts_tablet_manager.cc:1397] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250110 02:51:25.544378 2456 raft_consensus.cc:491] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.544472 2456 raft_consensus.cc:513] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.545022 2456 leader_election.cc:290] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.545034 1184 raft_consensus.cc:357] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.545231 1184 raft_consensus.cc:383] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.545284 1184 raft_consensus.cc:738] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.545471 1184 consensus_queue.cc:260] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.545823 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.545996 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.549089 2352 rpcz_store.cc:276] 0110 02:51:24.161916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.161949 (+ 33us) service_pool.cc:225] Handling call
0110 02:51:25.488322 (+1326373us) ts_tablet_manager.cc:1145] Deleting tablet da8f247c862e4cb68452e230857ee6b0
0110 02:51:25.488331 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.502037 (+ 13706us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.529529 (+ 27492us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.530763 (+ 1234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326255}
I20250110 02:51:25.549687 2377 raft_consensus.cc:491] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.549829 2377 raft_consensus.cc:513] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.550345 2377 leader_election.cc:290] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.550699 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet da8f247c862e4cb68452e230857ee6b0 (table default.Table_90e43feb19a343f6b3c9a20876ec8c23 [id=e2a6764b09814b9cb1367abb1a4f5ca9]) successfully deleted
I20250110 02:51:25.551542 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.551716 768 raft_consensus.cc:2463] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.551694 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.551893 1061 raft_consensus.cc:2463] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.552029 853 leader_election.cc:304] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.552438 2345 raft_consensus.cc:2798] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.552534 2345 raft_consensus.cc:491] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.552613 2345 raft_consensus.cc:3054] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.536964 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:25.532430 1188 tablet_bootstrap.cc:492] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.554584 1188 ts_tablet_manager.cc:1397] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.018s
I20250110 02:51:25.555336 1188 raft_consensus.cc:357] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.555557 1188 raft_consensus.cc:383] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.555660 1188 raft_consensus.cc:738] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.545542 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de67338a26f4febb5dd7611a72b4723" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.546245 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.556227 994 leader_election.cc:304] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:25.556365 709 leader_election.cc:343] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.556476 709 leader_election.cc:304] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.556572 2456 raft_consensus.cc:2743] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.549547 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 122) took 1387 ms. Trace:
I20250110 02:51:25.556834 2199 raft_consensus.cc:2743] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.556766 2352 rpcz_store.cc:276] 0110 02:51:24.162138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.162167 (+ 29us) service_pool.cc:225] Handling call
0110 02:51:25.549360 (+1387193us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:25.549369 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.549505 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1387102}
I20250110 02:51:25.557109 1188 consensus_queue.cc:260] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:25.557389 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e05a7554a14805bdcbd1824a5bd9d5 already in progress: creating tablet
I20250110 02:51:25.557705 2352 tablet_replica.cc:331] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.557945 2352 raft_consensus.cc:2238] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.555336 2345 raft_consensus.cc:513] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.559564 2345 leader_election.cc:290] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.560092 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:25.560235 1061 raft_consensus.cc:3054] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.560940 2199 raft_consensus.cc:491] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.561057 2199 raft_consensus.cc:513] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.561594 2199 leader_election.cc:290] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.542147 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:25.562469 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 85) took 1392 ms. Trace:
I20250110 02:51:25.562554 2357 rpcz_store.cc:276] 0110 02:51:24.170444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.170619 (+ 175us) service_pool.cc:225] Handling call
0110 02:51:25.534575 (+1363956us) ts_tablet_manager.cc:1145] Deleting tablet d761ef0d1e5542b1979c29bb2d579489
0110 02:51:25.534584 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.562450 (+ 27866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363469}
W20250110 02:51:25.562670 994 leader_election.cc:343] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.562798 1188 ts_tablet_manager.cc:1428] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250110 02:51:25.563066 1188 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.563071 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.563120 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9507a5283d8f4d07a293ba474c642298" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:25.563174 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
I20250110 02:51:25.563254 768 raft_consensus.cc:3054] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:25.563499 711 leader_election.cc:343] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.563793 1061 raft_consensus.cc:2463] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.564204 853 leader_election.cc:304] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:25.564544 2378 raft_consensus.cc:2798] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.564667 2352 raft_consensus.cc:2267] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.564675 2378 raft_consensus.cc:695] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:25.564970 2378 consensus_queue.cc:237] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.565227 2352 ts_tablet_manager.cc:1905] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.565642 768 raft_consensus.cc:2463] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.565946 2199 raft_consensus.cc:491] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.566092 2199 raft_consensus.cc:513] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.566619 2199 leader_election.cc:290] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.567191 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.567636 711 leader_election.cc:343] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.564980 1188 tablet_bootstrap.cc:654] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.570057 1181 raft_consensus.cc:357] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.570436 1181 raft_consensus.cc:383] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.570556 1181 raft_consensus.cc:738] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.570993 1181 consensus_queue.cc:260] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.571564 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.572032 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.572881 709 leader_election.cc:343] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.573019 709 leader_election.cc:304] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:25.573470 709 leader_election.cc:343] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.573567 709 leader_election.cc:304] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.574657 2199 raft_consensus.cc:2743] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.574785 2199 raft_consensus.cc:2743] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.578861 2352 ts_tablet_manager.cc:1918] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.545930 1184 ts_tablet_manager.cc:1428] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:25.586561 2345 raft_consensus.cc:491] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.587242 2345 raft_consensus.cc:513] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.588116 2345 leader_election.cc:290] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.589498 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.588374 2352 log.cc:1198] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/0127de45264f4660b843734a530eb310
I20250110 02:51:25.589715 764 raft_consensus.cc:2388] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:25.590111 2352 ts_tablet_manager.cc:1939] T 0127de45264f4660b843734a530eb310 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.590468 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 123) took 1428 ms. Trace:
I20250110 02:51:25.593773 2352 rpcz_store.cc:276] 0110 02:51:24.162324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.162352 (+ 28us) service_pool.cc:225] Handling call
0110 02:51:25.557041 (+1394689us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:25.557621 (+ 580us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.568552 (+ 10931us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.578708 (+ 10156us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.590413 (+ 11705us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6521,"tablet-delete.queue_time_us":1394569}
I20250110 02:51:25.594202 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 0127de45264f4660b843734a530eb310 (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:25.594910 2352 tablet_replica.cc:331] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.600092 2352 raft_consensus.cc:2238] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.600502 2352 raft_consensus.cc:2267] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.603327 2357 tablet_replica.cc:331] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.603530 2357 raft_consensus.cc:2238] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.603672 2357 raft_consensus.cc:2267] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.603965 1184 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.604226 2357 ts_tablet_manager.cc:1905] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.588723 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edb4e06e40c43438882e1eb622d788e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.589005 2456 raft_consensus.cc:491] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.605530 1184 tablet_bootstrap.cc:654] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.605548 2456 raft_consensus.cc:513] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.606093 2456 leader_election.cc:290] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
W20250110 02:51:25.606925 996 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:25.607518 1181 ts_tablet_manager.cc:1428] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20250110 02:51:25.608081 1181 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.611268 1181 tablet_bootstrap.cc:654] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.611619 2357 ts_tablet_manager.cc:1918] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.611721 2357 log.cc:1198] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:25.612007 2357 ts_tablet_manager.cc:1939] T cdb726d2c9a04963b92a3fb625cf84f3 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.612751 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 61) took 1466 ms. Trace:
I20250110 02:51:25.620111 1184 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.620244 1184 ts_tablet_manager.cc:1397] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20250110 02:51:25.620992 1184 raft_consensus.cc:357] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.621155 1184 raft_consensus.cc:383] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.621199 1184 raft_consensus.cc:738] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:25.627655 994 consensus_peers.cc:487] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eff9b10c419242cf85ef1b708bd5a862. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.605616 853 leader_election.cc:304] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:25.631624 2377 raft_consensus.cc:3054] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.634058 2377 raft_consensus.cc:2743] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.606523 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302af84fd8374ffa96a7503b54849536" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.635720 994 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.635839 994 leader_election.cc:304] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:25.636166 2432 raft_consensus.cc:2743] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.638969 2352 ts_tablet_manager.cc:1905] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.643801 2224 ts_tablet_manager.cc:1918] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.643918 2224 log.cc:1198] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:25.644237 2224 ts_tablet_manager.cc:1939] T f66603a0ee8c4082807c5a6aabe7e339 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.644918 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 91) took 2489 ms. Trace:
I20250110 02:51:25.645004 2224 rpcz_store.cc:276] 0110 02:51:23.155671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.155723 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:25.543110 (+2387387us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:25.543120 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.591254 (+ 48134us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.643716 (+ 52462us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.644899 (+ 1183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367937}
I20250110 02:51:25.645294 2224 tablet_replica.cc:331] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.645455 2224 raft_consensus.cc:2238] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.645491 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f66603a0ee8c4082807c5a6aabe7e339 (table default.Table_7ab0055319bd4962a0267ce290806e0f [id=8488c24625ca46ba8f0d2405ab565feb]) successfully deleted
I20250110 02:51:25.645709 2224 raft_consensus.cc:2267] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.646171 2224 ts_tablet_manager.cc:1905] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.646507 1184 consensus_queue.cc:260] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.647025 1184 ts_tablet_manager.cc:1428] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250110 02:51:25.647301 1184 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.648924 1184 tablet_bootstrap.cc:654] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.650426 1188 tablet_bootstrap.cc:492] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.650583 1188 ts_tablet_manager.cc:1397] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.011s
I20250110 02:51:25.651311 1188 raft_consensus.cc:357] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.651515 1188 raft_consensus.cc:383] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.651587 1188 raft_consensus.cc:738] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.651795 1188 consensus_queue.cc:260] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:25.664392 711 consensus_peers.cc:487] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e79175853474cbea951f5a5686bf630. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.672211 2378 raft_consensus.cc:491] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.672353 2378 raft_consensus.cc:513] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.672588 1188 ts_tablet_manager.cc:1428] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250110 02:51:25.672976 2378 leader_election.cc:290] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.673657 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:25.673830 764 raft_consensus.cc:2463] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:25.674257 853 leader_election.cc:304] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:25.674747 2378 raft_consensus.cc:2798] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.674755 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.674852 2378 raft_consensus.cc:491] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.674923 2378 raft_consensus.cc:3054] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:25.675195 853 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.677874 2378 raft_consensus.cc:513] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.679032 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
W20250110 02:51:25.679625 853 leader_election.cc:343] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.680011 1188 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.680935 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:25.681099 764 raft_consensus.cc:3054] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.681538 1188 tablet_bootstrap.cc:654] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.683699 764 raft_consensus.cc:2463] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:25.683874 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:25.684231 853 leader_election.cc:304] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.684507 2377 raft_consensus.cc:2798] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.684648 2377 raft_consensus.cc:695] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:25.684890 2377 consensus_queue.cc:237] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.687387 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:25.691738 2378 leader_election.cc:290] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:25.695749 2559 tablet.cc:1721] T 3e4d2bb0b992474d8e5d21abaca5411c P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:25.699376 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:25.699716 2561 tablet.cc:1721] T 3e4d2bb0b992474d8e5d21abaca5411c P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:25.698017 2560 tablet.cc:1721] T 3e4d2bb0b992474d8e5d21abaca5411c P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:25.612850 2357 rpcz_store.cc:276] 0110 02:51:24.146022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.178492 (+ 32470us) service_pool.cc:225] Handling call
0110 02:51:25.562795 (+1384303us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:25.603024 (+ 40229us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.608844 (+ 5820us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.611537 (+ 2693us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.612728 (+ 1191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1383331}
I20250110 02:51:25.706166 1188 tablet_bootstrap.cc:492] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.706269 1188 ts_tablet_manager.cc:1397] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20250110 02:51:25.706575 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:25.706624 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet cdb726d2c9a04963b92a3fb625cf84f3 (table default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]) successfully deleted
I20250110 02:51:25.706810 2357 tablet_replica.cc:331] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.707006 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:25.707031 2357 raft_consensus.cc:2238] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.707204 2357 raft_consensus.cc:2267] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.707460 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:25.707778 2357 ts_tablet_manager.cc:1905] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.708037 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:25.708303 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:25.708837 1188 raft_consensus.cc:357] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.709859 2549 raft_consensus.cc:491] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.709985 2549 raft_consensus.cc:513] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.710677 2549 leader_election.cc:290] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.711009 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.711342 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
W20250110 02:51:25.711551 711 leader_election.cc:343] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.711627 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:25.711671 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.712085 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:25.711843 1188 raft_consensus.cc:383] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.712250 1188 raft_consensus.cc:738] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:25.712327 709 leader_election.cc:343] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.712437 709 leader_election.cc:304] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.712651 2549 raft_consensus.cc:2743] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.712926 1188 consensus_queue.cc:260] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.715760 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:25.716181 1188 ts_tablet_manager.cc:1428] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:25.716759 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:25.717168 1188 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.718114 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:25.719691 1188 tablet_bootstrap.cc:654] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.720145 1439 catalog_manager.cc:5526] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:25.721374 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fa40c9c3f0ef447294b7c1c5f29ba06e (table default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9]). Expected version 3 got 0
I20250110 02:51:25.721949 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:25.722582 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 598bc151d0c14fc6bb28523c535d9366 (table default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989]). Expected version 1 got 0
I20250110 02:51:25.723996 2549 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.724130 2549 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.724690 2549 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.725034 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.725513 711 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.725665 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.726158 709 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.726272 709 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.726655 2549 raft_consensus.cc:2743] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.730778 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:25.732345 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:25.737380 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:25.738176 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:25.739364 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_09da00340dfa4ef59164793be294d5df [id=0692620395c14593803592e6aade6af3]). Expected version 2 got 1
I20250110 02:51:25.740139 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:25.741478 2224 ts_tablet_manager.cc:1918] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.741582 2224 log.cc:1198] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/0127de45264f4660b843734a530eb310
I20250110 02:51:25.741833 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:25.741901 2224 ts_tablet_manager.cc:1939] T 0127de45264f4660b843734a530eb310 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.742321 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:25.743041 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:25.747270 1184 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.747450 1184 ts_tablet_manager.cc:1397] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.001s
I20250110 02:51:25.748021 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:25.748265 1184 raft_consensus.cc:357] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.748540 1184 raft_consensus.cc:383] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.748633 1184 raft_consensus.cc:738] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.749573 1184 consensus_queue.cc:260] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.750145 1184 ts_tablet_manager.cc:1428] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:25.751497 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 92) took 2482 ms. Trace:
I20250110 02:51:25.751587 2224 rpcz_store.cc:276] 0110 02:51:23.268776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.268853 (+ 77us) service_pool.cc:225] Handling call
0110 02:51:25.645254 (+2376401us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:25.645263 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.688502 (+ 43239us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.741391 (+ 52889us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.751476 (+ 10085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376114}
I20250110 02:51:25.751940 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 94) took 2482 ms. Trace:
I20250110 02:51:25.752014 2224 rpcz_store.cc:276] 0110 02:51:23.269900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.269957 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:25.751836 (+2481879us) ts_tablet_manager.cc:1145] Deleting tablet c5601ca24b8f428e8a51cbeec3664901
0110 02:51:25.751844 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.751926 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481657}
I20250110 02:51:25.752110 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 0127de45264f4660b843734a530eb310 (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:25.752236 2224 tablet_replica.cc:331] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.752384 2224 raft_consensus.cc:2238] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.752514 2224 raft_consensus.cc:2267] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:25.752545 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5601ca24b8f428e8a51cbeec3664901 already in progress: creating tablet
I20250110 02:51:25.752966 2224 ts_tablet_manager.cc:1905] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.756771 1188 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.756916 1188 ts_tablet_manager.cc:1397] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.002s
I20250110 02:51:25.757567 1188 raft_consensus.cc:357] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.757737 1188 raft_consensus.cc:383] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.757788 1188 raft_consensus.cc:738] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.758299 2224 ts_tablet_manager.cc:1918] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.759171 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:25.759600 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:25.759996 2357 ts_tablet_manager.cc:1918] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.760113 2357 log.cc:1198] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b375a932156d4abb908dbf327689b0f9
I20250110 02:51:25.760460 2357 ts_tablet_manager.cc:1939] T b375a932156d4abb908dbf327689b0f9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.760761 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 87) took 1568 ms. Trace:
I20250110 02:51:25.760839 2357 rpcz_store.cc:276] 0110 02:51:24.192584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.192805 (+ 221us) service_pool.cc:225] Handling call
0110 02:51:25.706763 (+1513958us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:25.706774 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.756737 (+ 49963us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.759891 (+ 3154us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.760745 (+ 854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513741}
I20250110 02:51:25.761260 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b375a932156d4abb908dbf327689b0f9 (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
I20250110 02:51:25.760660 1188 consensus_queue.cc:260] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.763739 2357 tablet_replica.cc:331] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.763911 1188 ts_tablet_manager.cc:1428] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:25.764178 1188 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.765367 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:25.765544 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:25.765895 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:25.777276 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:25.777562 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:25.758634 1181 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.777711 2224 log.cc:1198] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:25.777760 1181 ts_tablet_manager.cc:1397] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.170s user 0.006s sys 0.012s
I20250110 02:51:25.778134 2224 ts_tablet_manager.cc:1939] T d10ace37b96844328ccdb821dad3c5ea P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.779284 1181 raft_consensus.cc:357] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.779456 1181 raft_consensus.cc:383] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.779506 1181 raft_consensus.cc:738] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.779680 1181 consensus_queue.cc:260] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.780164 1181 ts_tablet_manager.cc:1428] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:25.780449 1181 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.781978 1181 tablet_bootstrap.cc:654] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.783566 1188 tablet_bootstrap.cc:654] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.784220 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 95) took 2421 ms. Trace:
I20250110 02:51:25.784307 2224 rpcz_store.cc:276] 0110 02:51:23.362536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.362930 (+ 394us) service_pool.cc:225] Handling call
0110 02:51:25.752202 (+2389272us) ts_tablet_manager.cc:1145] Deleting tablet d10ace37b96844328ccdb821dad3c5ea
0110 02:51:25.752209 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.755654 (+ 3445us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.758230 (+ 2576us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.784172 (+ 25942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376801}
I20250110 02:51:25.784694 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d10ace37b96844328ccdb821dad3c5ea (table default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]) successfully deleted
I20250110 02:51:25.785193 2224 tablet_replica.cc:331] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.790627 2357 raft_consensus.cc:2238] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.790843 2357 raft_consensus.cc:2267] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.791468 2357 ts_tablet_manager.cc:1905] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.792814 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:25.794358 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:25.797072 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:25.801977 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:25.804387 1188 tablet_bootstrap.cc:492] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.804562 1188 ts_tablet_manager.cc:1397] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250110 02:51:25.805289 1188 raft_consensus.cc:357] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.805526 1188 raft_consensus.cc:383] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.806460 1188 raft_consensus.cc:738] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.806735 1188 consensus_queue.cc:260] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.807260 1188 ts_tablet_manager.cc:1428] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:25.818456 1457 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 104) took 1166 ms. Trace:
I20250110 02:51:25.818580 1457 rpcz_store.cc:276] 0110 02:51:24.652477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.414315 (+761838us) service_pool.cc:225] Handling call
0110 02:51:25.818418 (+404103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:25.828791 2357 ts_tablet_manager.cc:1918] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.828927 2357 log.cc:1198] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:25.828451 2352 ts_tablet_manager.cc:1918] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.829294 2357 ts_tablet_manager.cc:1939] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.829612 2352 log.cc:1198] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:25.830072 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 89) took 1635 ms. Trace:
I20250110 02:51:25.831669 2352 ts_tablet_manager.cc:1939] T f66603a0ee8c4082807c5a6aabe7e339 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:25.833688 2199 raft_consensus.cc:491] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.833792 2199 raft_consensus.cc:513] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.834301 2199 leader_election.cc:290] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.834743 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.834939 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.835208 917 raft_consensus.cc:2463] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:25.835441 711 leader_election.cc:343] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.835922 709 leader_election.cc:304] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.836513 2199 raft_consensus.cc:2798] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:25.836580 2199 raft_consensus.cc:491] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:25.836627 2199 raft_consensus.cc:3054] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.836967 1188 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.838609 1188 tablet_bootstrap.cc:654] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.839457 2199 raft_consensus.cc:513] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.840550 2199 leader_election.cc:290] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.840943 1181 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.841104 1181 ts_tablet_manager.cc:1397] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.002s
W20250110 02:51:25.841325 711 consensus_peers.cc:487] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1390f867a3a8436092d1920daff4c5b0. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:25.841334 2199 raft_consensus.cc:491] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.841513 2199 raft_consensus.cc:513] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.841627 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:25.841955 1181 raft_consensus.cc:357] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:25.842097 711 leader_election.cc:343] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.842216 1181 raft_consensus.cc:383] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.842314 1181 raft_consensus.cc:738] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.842458 2199 leader_election.cc:290] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.842574 1181 consensus_queue.cc:260] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.842789 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0b2ec1d5814144bc58261e6c6e8d35" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.842949 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0b2ec1d5814144bc58261e6c6e8d35" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.843196 1181 ts_tablet_manager.cc:1428] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:25.843358 711 leader_election.cc:343] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:25.843509 709 leader_election.cc:343] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.843595 709 leader_election.cc:304] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.840451 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e25f9643ac0476a90c93fb9aeb921a3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:25.844152 913 raft_consensus.cc:3054] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:25.830165 2357 rpcz_store.cc:276] 0110 02:51:24.194831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.194875 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:25.763695 (+1568820us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:25.763703 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.794930 (+ 31227us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.828670 (+ 33740us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.830042 (+ 1372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1568647}
I20250110 02:51:25.846869 2549 raft_consensus.cc:2743] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.847090 2357 tablet_replica.cc:331] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.847132 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
I20250110 02:51:25.846923 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 124) took 1684 ms. Trace:
I20250110 02:51:25.847254 2357 raft_consensus.cc:2238] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.847369 2357 raft_consensus.cc:2267] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.847868 2357 ts_tablet_manager.cc:1905] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.848013 913 raft_consensus.cc:2463] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:25.848351 709 leader_election.cc:304] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.847291 2352 rpcz_store.cc:276] 0110 02:51:24.162554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.162587 (+ 33us) service_pool.cc:225] Handling call
0110 02:51:25.594785 (+1432198us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:25.594798 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.641779 (+ 46981us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.827829 (+186050us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.846911 (+ 19082us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3684,"tablet-delete.queue_time_us":1432089}
I20250110 02:51:25.848662 2549 raft_consensus.cc:2798] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:25.848765 2549 raft_consensus.cc:695] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:25.848969 2549 consensus_queue.cc:237] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.849092 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f66603a0ee8c4082807c5a6aabe7e339 (table default.Table_7ab0055319bd4962a0267ce290806e0f [id=8488c24625ca46ba8f0d2405ab565feb]) successfully deleted
I20250110 02:51:25.850487 2352 tablet_replica.cc:331] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:25.852196 2352 raft_consensus.cc:2238] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.853166 2352 raft_consensus.cc:2267] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.854106 2224 raft_consensus.cc:2238] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.854439 1181 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.855952 1184 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.856472 1181 tablet_bootstrap.cc:654] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.857491 2224 raft_consensus.cc:2267] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.858062 2224 ts_tablet_manager.cc:1905] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.871209 1184 tablet_bootstrap.cc:654] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.875056 2224 ts_tablet_manager.cc:1918] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.875332 2224 log.cc:1198] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8c3ce95413df40369ee599176f91b892
I20250110 02:51:25.876144 2224 ts_tablet_manager.cc:1939] T 8c3ce95413df40369ee599176f91b892 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.876973 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 99) took 2501 ms. Trace:
I20250110 02:51:25.879009 2224 rpcz_store.cc:276] 0110 02:51:23.375840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.375953 (+ 113us) service_pool.cc:225] Handling call
0110 02:51:25.785114 (+2409161us) ts_tablet_manager.cc:1145] Deleting tablet 8c3ce95413df40369ee599176f91b892
0110 02:51:25.785124 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.860903 (+ 75779us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.873336 (+ 12433us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.876957 (+ 3621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408846}
I20250110 02:51:25.878803 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2d48031d9dc54627ae45e5538a7290d7 (DEFAULT_TABLE table=default.Table_83db50070a234025b9d36aa4b1fe952a [id=c1a619572bd4484abceefbd5602fc07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:25.878628 2352 ts_tablet_manager.cc:1905] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.879266 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 2d48031d9dc54627ae45e5538a7290d7 (DEFAULT_TABLE table=default.Table_83db50070a234025b9d36aa4b1fe952a [id=c1a619572bd4484abceefbd5602fc07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:25.880005 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 97) took 2505 ms. Trace:
I20250110 02:51:25.880506 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8c3ce95413df40369ee599176f91b892 (table default.Table_a95ba565dbf940f79ed6f8b2eff1e74b [id=0743ba60cabd49c7a2c5230109d764eb]) successfully deleted
I20250110 02:51:25.880458 2224 rpcz_store.cc:276] 0110 02:51:23.374678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.375132 (+ 454us) service_pool.cc:225] Handling call
0110 02:51:25.879884 (+2504752us) ts_tablet_manager.cc:1145] Deleting tablet d761ef0d1e5542b1979c29bb2d579489
0110 02:51:25.879899 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.879991 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":2502272}
I20250110 02:51:25.880743 2224 tablet_replica.cc:331] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:25.881088 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d761ef0d1e5542b1979c29bb2d579489 already in progress: creating tablet
I20250110 02:51:25.880360 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d48031d9dc54627ae45e5538a7290d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:25.882862 2224 raft_consensus.cc:2238] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.883002 2224 raft_consensus.cc:2267] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.883437 2224 ts_tablet_manager.cc:1905] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.881913 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2d48031d9dc54627ae45e5538a7290d7 (DEFAULT_TABLE table=default.Table_83db50070a234025b9d36aa4b1fe952a [id=c1a619572bd4484abceefbd5602fc07c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:25.890969 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d48031d9dc54627ae45e5538a7290d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:25.894492 2357 ts_tablet_manager.cc:1918] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:25.894603 2357 log.cc:1198] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:25.894928 2357 ts_tablet_manager.cc:1939] T 8f15078fc22f4b32a48c7190b36e2192 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.895254 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 86) took 1702 ms. Trace:
I20250110 02:51:25.895331 2357 rpcz_store.cc:276] 0110 02:51:24.192464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.195101 (+ 2637us) service_pool.cc:225] Handling call
0110 02:51:25.847032 (+1651931us) ts_tablet_manager.cc:1145] Deleting tablet 8f15078fc22f4b32a48c7190b36e2192
0110 02:51:25.847043 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.861158 (+ 14115us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.894416 (+ 33258us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.895235 (+ 819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651009}
I20250110 02:51:25.895732 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 91) took 1698 ms. Trace:
I20250110 02:51:25.895802 2357 rpcz_store.cc:276] 0110 02:51:24.197339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.197382 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:25.895620 (+1698238us) ts_tablet_manager.cc:1145] Deleting tablet a7397e41f7374e9fb1621dbcf30af703
0110 02:51:25.895630 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.895716 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698100}
I20250110 02:51:25.896083 2357 tablet_replica.cc:331] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.896260 2357 raft_consensus.cc:2238] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.896587 2357 raft_consensus.cc:2267] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.897146 2357 ts_tablet_manager.cc:1905] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.897527 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8f15078fc22f4b32a48c7190b36e2192 (table default.Table_5c647fe509fd45eebc5796422650b5d8 [id=510ab81389564eb4af6f119bb0e6f701]) successfully deleted
W20250110 02:51:25.897787 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a7397e41f7374e9fb1621dbcf30af703 because the tablet was not found. No further retry: Not found: Tablet not found: a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:25.903143 1181 tablet_bootstrap.cc:492] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.903302 1181 ts_tablet_manager.cc:1397] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.007s
I20250110 02:51:25.903995 1181 raft_consensus.cc:357] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.904217 1181 raft_consensus.cc:383] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.904294 1181 raft_consensus.cc:738] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.904495 1181 consensus_queue.cc:260] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.904839 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:25.905643 2224 ts_tablet_manager.cc:1918] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.905094 1181 ts_tablet_manager.cc:1428] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:25.905884 2224 log.cc:1198] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:25.906414 2224 ts_tablet_manager.cc:1939] T 5e183122d13a4c38988bf2fe6f7a3694 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.906728 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 103) took 2529 ms. Trace:
I20250110 02:51:25.906847 2224 rpcz_store.cc:276] 0110 02:51:23.376856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.376963 (+ 107us) service_pool.cc:225] Handling call
0110 02:51:25.880709 (+2503746us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:25.880717 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.886047 (+ 5330us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.905551 (+ 19504us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.906713 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502954}
I20250110 02:51:25.906999 1184 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.907120 1184 ts_tablet_manager.cc:1397] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
I20250110 02:51:25.907335 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 5e183122d13a4c38988bf2fe6f7a3694 (table default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]) successfully deleted
I20250110 02:51:25.907706 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 102) took 2530 ms. Trace:
I20250110 02:51:25.907814 2224 rpcz_store.cc:276] 0110 02:51:23.376779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.376920 (+ 141us) service_pool.cc:225] Handling call
0110 02:51:25.907601 (+2530681us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:25.907609 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.907691 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529586}
I20250110 02:51:25.907876 1184 raft_consensus.cc:357] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.908267 1184 raft_consensus.cc:383] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:25.908329 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452223af5f6a4836a0f582d394a9453e already in progress: creating tablet
I20250110 02:51:25.908375 1184 raft_consensus.cc:738] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.908639 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d48031d9dc54627ae45e5538a7290d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:25.908696 1184 consensus_queue.cc:260] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.909179 1184 ts_tablet_manager.cc:1428] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:25.912582 2345 consensus_queue.cc:1035] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:25.915370 2549 raft_consensus.cc:491] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.915489 2549 raft_consensus.cc:513] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.916288 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:25.916695 709 leader_election.cc:343] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.916908 2549 leader_election.cc:290] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.917234 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:25.917676 711 leader_election.cc:343] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.917768 711 leader_election.cc:304] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.918080 2549 raft_consensus.cc:2743] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.920048 1184 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:25.921476 1184 tablet_bootstrap.cc:654] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.930552 2224 tablet_replica.cc:331] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.930814 2224 raft_consensus.cc:2238] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.931226 2224 raft_consensus.cc:2267] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.931715 1188 tablet_bootstrap.cc:492] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:25.931931 1188 ts_tablet_manager.cc:1397] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.002s
I20250110 02:51:25.931947 2224 ts_tablet_manager.cc:1905] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.932873 1188 raft_consensus.cc:357] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.933112 1188 raft_consensus.cc:383] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.933192 1188 raft_consensus.cc:738] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.933822 1188 consensus_queue.cc:260] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:25.934549 1188 ts_tablet_manager.cc:1428] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:25.940411 2524 consensus_queue.cc:1035] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:25.941601 2224 ts_tablet_manager.cc:1918] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.949733 2224 log.cc:1198] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:25.953346 2224 ts_tablet_manager.cc:1939] T dad0d561a8ef498ba13bc813d4cd82fa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.953658 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 96) took 2579 ms. Trace:
I20250110 02:51:25.953776 2224 rpcz_store.cc:276] 0110 02:51:23.374606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.375076 (+ 470us) service_pool.cc:225] Handling call
0110 02:51:25.930500 (+2555424us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:25.930514 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.935576 (+ 5062us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.941524 (+ 5948us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.953645 (+ 12121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2550965}
I20250110 02:51:25.954279 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet dad0d561a8ef498ba13bc813d4cd82fa (table default.Table_5bfa749055ab45479cf96da68e537ece [id=488d922a6b604812af65ff6b4d9c6606]) successfully deleted
I20250110 02:51:25.954679 2224 tablet_replica.cc:331] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:25.954990 2224 raft_consensus.cc:2238] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:25.955348 2224 raft_consensus.cc:2267] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:25.955842 2224 ts_tablet_manager.cc:1905] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:25.954957 1188 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:25.958696 1181 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:25.960886 1181 tablet_bootstrap.cc:654] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.964130 1184 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:25.964277 1184 ts_tablet_manager.cc:1397] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.001s
I20250110 02:51:25.964558 2549 raft_consensus.cc:491] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:25.964681 2549 raft_consensus.cc:513] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.964982 1184 raft_consensus.cc:357] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.965137 1184 raft_consensus.cc:383] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.965222 1184 raft_consensus.cc:738] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.965272 2549 leader_election.cc:290] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:25.965396 1184 consensus_queue.cc:260] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.965657 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:25.965751 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:25.965893 1184 ts_tablet_manager.cc:1428] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:25.966130 1184 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:25.966171 711 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:25.966297 709 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:25.966423 709 leader_election.cc:304] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:25.966805 2199 raft_consensus.cc:2743] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:25.972460 2224 ts_tablet_manager.cc:1918] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.980091 1181 tablet_bootstrap.cc:492] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:25.980304 1181 ts_tablet_manager.cc:1397] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.009s
I20250110 02:51:25.981743 1181 raft_consensus.cc:357] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.981971 1181 raft_consensus.cc:383] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:25.982059 1181 raft_consensus.cc:738] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:25.982312 1181 consensus_queue.cc:260] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:25.983723 1184 tablet_bootstrap.cc:654] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:25.986460 1181 ts_tablet_manager.cc:1428] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:25.986627 2224 log.cc:1198] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8f4778347f054b86af697dbe3b287263
I20250110 02:51:25.987053 2224 ts_tablet_manager.cc:1939] T 8f4778347f054b86af697dbe3b287263 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:25.990939 2357 ts_tablet_manager.cc:1918] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:25.991798 2357 log.cc:1198] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:25.992250 2357 ts_tablet_manager.cc:1939] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:25.992609 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 90) took 1795 ms. Trace:
I20250110 02:51:25.992749 2357 rpcz_store.cc:276] 0110 02:51:24.197235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.197646 (+ 411us) service_pool.cc:225] Handling call
0110 02:51:25.896040 (+1698394us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:25.896050 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.967303 (+ 71253us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.990825 (+ 23522us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.992592 (+ 1767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698181}
I20250110 02:51:25.993336 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]) successfully deleted
I20250110 02:51:25.993922 2357 tablet_replica.cc:331] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:25.994182 2357 raft_consensus.cc:2238] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:25.996893 1061 raft_consensus.cc:3054] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.008044 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:26.008438 2357 raft_consensus.cc:2267] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.008654 1188 tablet_bootstrap.cc:654] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.009344 2357 ts_tablet_manager.cc:1905] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.010619 1181 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.011188 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 100) took 2635 ms. Trace:
I20250110 02:51:26.011260 2224 rpcz_store.cc:276] 0110 02:51:23.375907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.376100 (+ 193us) service_pool.cc:225] Handling call
0110 02:51:25.954638 (+2578538us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:25.954646 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.969790 (+ 15144us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:25.972390 (+ 2600us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.011172 (+ 38782us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574536}
I20250110 02:51:26.011600 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 101) took 2634 ms. Trace:
I20250110 02:51:26.011672 2224 rpcz_store.cc:276] 0110 02:51:23.376676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.376983 (+ 307us) service_pool.cc:225] Handling call
0110 02:51:26.011503 (+2634520us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:26.011511 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.011588 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2631261}
I20250110 02:51:26.011934 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 105) took 2633 ms. Trace:
I20250110 02:51:26.011991 2224 rpcz_store.cc:276] 0110 02:51:23.378452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.378759 (+ 307us) service_pool.cc:225] Handling call
0110 02:51:26.011843 (+2633084us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:26.011850 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.011923 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2631461}
I20250110 02:51:26.012197 2224 tablet_replica.cc:331] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.012356 2224 raft_consensus.cc:2238] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.012501 2224 raft_consensus.cc:2267] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.012934 2224 ts_tablet_manager.cc:1905] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.023375 1184 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.023548 1184 ts_tablet_manager.cc:1397] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250110 02:51:26.024189 2224 ts_tablet_manager.cc:1918] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.024322 2224 log.cc:1198] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:26.024358 1184 raft_consensus.cc:357] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.024629 1184 raft_consensus.cc:383] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.024655 2224 ts_tablet_manager.cc:1939] T 09f01daf4c67462ba75e1d0cf56cef12 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.024744 1184 raft_consensus.cc:738] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.024940 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 104) took 2646 ms. Trace:
I20250110 02:51:26.025018 2224 rpcz_store.cc:276] 0110 02:51:23.378308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.378783 (+ 475us) service_pool.cc:225] Handling call
0110 02:51:26.012164 (+2633381us) ts_tablet_manager.cc:1145] Deleting tablet 09f01daf4c67462ba75e1d0cf56cef12
0110 02:51:26.012171 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.019731 (+ 7560us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.024110 (+ 4379us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.024919 (+ 809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2631664}
I20250110 02:51:26.025081 1184 consensus_queue.cc:260] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.025465 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 98) took 2649 ms. Trace:
I20250110 02:51:26.025705 1184 ts_tablet_manager.cc:1428] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.017823 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8f4778347f054b86af697dbe3b287263 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]) successfully deleted
W20250110 02:51:26.026099 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
W20250110 02:51:26.026325 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:26.026561 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 09f01daf4c67462ba75e1d0cf56cef12 (table default.Table_72de93e5a855471e85416b410edf348d [id=9c8eec49718f4b089953cb1ab7b7a57a]) successfully deleted
I20250110 02:51:26.027338 1184 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.028334 2352 ts_tablet_manager.cc:1918] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.028570 2352 log.cc:1198] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:26.029163 2352 ts_tablet_manager.cc:1939] T d10ace37b96844328ccdb821dad3c5ea P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.029335 1184 tablet_bootstrap.cc:654] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.030341 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 125) took 1867 ms. Trace:
I20250110 02:51:26.031013 2352 rpcz_store.cc:276] 0110 02:51:24.162777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.162814 (+ 37us) service_pool.cc:225] Handling call
0110 02:51:25.849936 (+1687122us) ts_tablet_manager.cc:1145] Deleting tablet d10ace37b96844328ccdb821dad3c5ea
0110 02:51:25.849946 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:25.900561 (+ 50615us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.028202 (+127641us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.030322 (+ 2120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1167,"tablet-delete.queue_time_us":1686992}
I20250110 02:51:26.031668 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d10ace37b96844328ccdb821dad3c5ea (table default.Table_5d9b00106d6148598a57409f1df9ad37 [id=a0a5eebef8964cc4a5569a725b1d13aa]) successfully deleted
I20250110 02:51:26.034926 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:26.035867 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:26.036213 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:26.036635 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:26.036823 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:26.037086 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:26.037246 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:26.037510 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:26.037685 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:26.037866 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:26.046452 1188 tablet_bootstrap.cc:492] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.025581 2224 rpcz_store.cc:276] 0110 02:51:23.375767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.376140 (+ 373us) service_pool.cc:225] Handling call
0110 02:51:26.025365 (+2649225us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:26.025373 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.025451 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643876}
I20250110 02:51:26.046658 1188 ts_tablet_manager.cc:1397] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
W20250110 02:51:26.046970 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
I20250110 02:51:26.047515 1188 raft_consensus.cc:357] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.047762 1188 raft_consensus.cc:383] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.047897 1188 raft_consensus.cc:738] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.048192 1188 consensus_queue.cc:260] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.048872 1181 tablet_bootstrap.cc:654] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.049052 1188 ts_tablet_manager.cc:1428] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:26.051265 1188 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.052518 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 107) took 2673 ms. Trace:
I20250110 02:51:26.052663 1188 tablet_bootstrap.cc:654] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.053877 2224 rpcz_store.cc:276] 0110 02:51:23.378659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.378732 (+ 73us) service_pool.cc:225] Handling call
0110 02:51:26.052406 (+2673674us) ts_tablet_manager.cc:1145] Deleting tablet 3fb805b3e537466ab7fe0185421ad5aa
0110 02:51:26.052414 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.052501 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2669031}
I20250110 02:51:26.054770 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 117) took 2669 ms. Trace:
I20250110 02:51:26.055130 2224 rpcz_store.cc:276] 0110 02:51:23.384861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.384907 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:26.054669 (+2669762us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:26.054678 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.054756 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2669613}
I20250110 02:51:26.055752 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 116) took 2671 ms. Trace:
W20250110 02:51:26.056255 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
W20250110 02:51:26.056514 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d4f03a8c66446ea7c6d178cc7ef80a already in progress: creating tablet
I20250110 02:51:26.056849 2352 tablet_replica.cc:331] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.057111 2352 raft_consensus.cc:2238] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.057397 2352 raft_consensus.cc:2267] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.038321 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:26.059151 2224 rpcz_store.cc:276] 0110 02:51:23.384193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.384574 (+ 381us) service_pool.cc:225] Handling call
0110 02:51:26.055661 (+2671087us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:26.055670 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.055737 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2667991}
I20250110 02:51:26.059384 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:26.059577 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:26.059870 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
W20250110 02:51:26.059844 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
I20250110 02:51:26.060182 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:26.060483 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 115) took 2676 ms. Trace:
I20250110 02:51:26.060688 2224 rpcz_store.cc:276] 0110 02:51:23.384112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.385552 (+ 1440us) service_pool.cc:225] Handling call
0110 02:51:26.060205 (+2674653us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:26.060213 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.060470 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672361}
I20250110 02:51:26.060781 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:26.060976 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
W20250110 02:51:26.061318 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4042c38de4969889f4821176fc7bb already in progress: creating tablet
I20250110 02:51:26.061542 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:26.061736 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:26.061977 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 113) took 2679 ms. Trace:
I20250110 02:51:26.061975 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:26.062153 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:26.063002 2432 raft_consensus.cc:491] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.063113 2432 raft_consensus.cc:513] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.063607 2432 leader_election.cc:290] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.064234 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.064386 913 raft_consensus.cc:2463] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.064652 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.064698 994 leader_election.cc:304] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.064821 764 raft_consensus.cc:2463] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.065018 2352 ts_tablet_manager.cc:1905] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.065517 2289 raft_consensus.cc:2798] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.065589 2289 raft_consensus.cc:491] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.065641 2289 raft_consensus.cc:3054] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.067833 2289 raft_consensus.cc:513] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.068375 2289 leader_election.cc:290] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.068522 2289 raft_consensus.cc:491] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.068660 2289 raft_consensus.cc:513] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.068902 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:26.069074 913 raft_consensus.cc:3054] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.069242 2289 leader_election.cc:290] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:26.069788 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.069947 917 raft_consensus.cc:2463] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.070286 994 leader_election.cc:304] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.070539 2289 raft_consensus.cc:2798] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.070595 2289 raft_consensus.cc:491] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.070642 2289 raft_consensus.cc:3054] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.071880 913 raft_consensus.cc:2463] T fb2272dfd0b14113afa3e13b38de6f8c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.072016 2357 ts_tablet_manager.cc:1918] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.072108 2357 log.cc:1198] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d9018955f3884824beb2e7336987fc1c
I20250110 02:51:26.072129 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2272dfd0b14113afa3e13b38de6f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.072227 994 leader_election.cc:304] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.072290 764 raft_consensus.cc:3054] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.072422 2357 ts_tablet_manager.cc:1939] T d9018955f3884824beb2e7336987fc1c P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.072450 2432 raft_consensus.cc:2798] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.072542 2432 raft_consensus.cc:695] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:26.072738 2432 consensus_queue.cc:237] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.074510 764 raft_consensus.cc:2463] T fb2272dfd0b14113afa3e13b38de6f8c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.074733 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.074819 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 93) took 1876 ms. Trace:
I20250110 02:51:26.074885 768 raft_consensus.cc:2463] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.075486 1188 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.075487 1184 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.075599 1184 ts_tablet_manager.cc:1397] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20250110 02:51:26.075640 1188 ts_tablet_manager.cc:1397] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20250110 02:51:26.076334 1184 raft_consensus.cc:357] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.076334 1188 raft_consensus.cc:357] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.076486 1188 raft_consensus.cc:383] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.076565 1184 raft_consensus.cc:383] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.076567 1188 raft_consensus.cc:738] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.076663 1184 raft_consensus.cc:738] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.076751 1188 consensus_queue.cc:260] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.076957 1184 consensus_queue.cc:260] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.077358 1188 ts_tablet_manager.cc:1428] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:26.077456 1184 ts_tablet_manager.cc:1428] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.077603 1188 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.077561 2289 raft_consensus.cc:513] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.077724 1184 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.078694 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.078852 768 raft_consensus.cc:3054] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.080245 2289 leader_election.cc:290] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:26.081039 768 raft_consensus.cc:2463] T 5a9d524fda0d4848b333df9e20a9f68b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.081549 996 leader_election.cc:304] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:26.081584 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a9d524fda0d4848b333df9e20a9f68b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:26.081703 913 raft_consensus.cc:3054] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.082065 2456 raft_consensus.cc:2798] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.082195 2456 raft_consensus.cc:695] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:26.082466 2478 raft_consensus.cc:491] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.082404 2456 consensus_queue.cc:237] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.082556 2478 raft_consensus.cc:513] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.082966 2478 leader_election.cc:290] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.083443 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de9cf980d9c4117b4ec3ca937cfcf01" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.083907 994 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.083963 913 raft_consensus.cc:2463] T 5a9d524fda0d4848b333df9e20a9f68b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.084568 1184 tablet_bootstrap.cc:654] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:26.084957 996 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:26.085098 996 leader_election.cc:304] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.085547 2478 raft_consensus.cc:2743] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.085666 1188 tablet_bootstrap.cc:654] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.086165 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:26.086475 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:26.086774 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:26.088116 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:26.074885 2357 rpcz_store.cc:276] 0110 02:51:24.198362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.198565 (+ 203us) service_pool.cc:225] Handling call
0110 02:51:25.993884 (+1795319us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:25.993892 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.046746 (+ 52854us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.071915 (+ 25169us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.074795 (+ 2880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1628,"tablet-delete.queue_time_us":1795139}
I20250110 02:51:26.091151 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d9018955f3884824beb2e7336987fc1c (table default.Table_570c693b36a44e53b70e27098df8de91 [id=adf485cc30d94278814a7d6ed6116302]) successfully deleted
I20250110 02:51:26.091533 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 94) took 1893 ms. Trace:
I20250110 02:51:26.091643 2357 rpcz_store.cc:276] 0110 02:51:24.198483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.198773 (+ 290us) service_pool.cc:225] Handling call
0110 02:51:26.091429 (+1892656us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:26.091437 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.091519 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1516,"tablet-delete.queue_time_us":1892535}
W20250110 02:51:26.092119 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:26.092406 2357 tablet_replica.cc:331] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.092602 2357 raft_consensus.cc:2238] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.092943 2357 raft_consensus.cc:2267] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.093446 2357 ts_tablet_manager.cc:1905] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.100052 1188 tablet_bootstrap.cc:492] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.100234 1188 ts_tablet_manager.cc:1397] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250110 02:51:26.101011 1188 raft_consensus.cc:357] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.101258 1188 raft_consensus.cc:383] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.101337 1188 raft_consensus.cc:738] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.101558 1188 consensus_queue.cc:260] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.101768 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:26.102104 1188 ts_tablet_manager.cc:1428] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:26.102447 1181 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.102638 1181 ts_tablet_manager.cc:1397] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.002s
I20250110 02:51:26.102870 2289 raft_consensus.cc:491] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.102988 2289 raft_consensus.cc:513] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.103528 2289 leader_election.cc:290] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.104040 1181 raft_consensus.cc:357] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.104317 1181 raft_consensus.cc:383] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.104418 1181 raft_consensus.cc:738] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.104640 1181 consensus_queue.cc:260] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.105237 1181 ts_tablet_manager.cc:1428] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.105495 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.105796 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c031451104d5a93a1672b4b1918c5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.105917 1188 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.105999 768 raft_consensus.cc:2388] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8899dd1aeb384e4ea4523f5fd225c888 in current term 1: Already voted for candidate 85f9597077934f1a9814d284ed06d13d in this term.
I20250110 02:51:26.106333 996 leader_election.cc:304] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.107314 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:26.107450 1188 tablet_bootstrap.cc:654] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.107977 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:26.108222 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:26.109383 1184 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.109560 1184 ts_tablet_manager.cc:1397] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250110 02:51:26.110666 1184 raft_consensus.cc:357] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.110932 1184 raft_consensus.cc:383] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.111027 1184 raft_consensus.cc:738] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.111299 1184 consensus_queue.cc:260] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.111944 1184 ts_tablet_manager.cc:1428] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.112179 1184 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.112529 2357 ts_tablet_manager.cc:1918] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.112620 2357 log.cc:1198] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:26.112900 2357 ts_tablet_manager.cc:1939] T 04e05a7554a14805bdcbd1824a5bd9d5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.113675 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 92) took 1915 ms. Trace:
I20250110 02:51:26.113765 2357 rpcz_store.cc:276] 0110 02:51:24.198250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.199491 (+ 1241us) service_pool.cc:225] Handling call
0110 02:51:26.092370 (+1892879us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:26.092378 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.096317 (+ 3939us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.112430 (+ 16113us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.113658 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":806,"tablet-delete.queue_time_us":1892745}
I20250110 02:51:26.114193 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 04e05a7554a14805bdcbd1824a5bd9d5 (table default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]) successfully deleted
I20250110 02:51:26.114511 2432 raft_consensus.cc:3054] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.116750 2432 raft_consensus.cc:2743] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.062192 2224 rpcz_store.cc:276] 0110 02:51:23.382807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.382858 (+ 51us) service_pool.cc:225] Handling call
0110 02:51:26.061580 (+2678722us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:26.061588 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.061962 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670350}
I20250110 02:51:26.118796 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 95) took 1918 ms. Trace:
I20250110 02:51:26.118865 2357 rpcz_store.cc:276] 0110 02:51:24.200088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.200150 (+ 62us) service_pool.cc:225] Handling call
0110 02:51:26.118564 (+1918414us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:26.118693 (+ 129us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.118781 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160,"tablet-delete.queue_time_us":1918268}
W20250110 02:51:26.119205 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:26.119808 2224 tablet_replica.cc:331] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.120978 2224 raft_consensus.cc:2238] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.121205 2224 raft_consensus.cc:2267] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.121804 2224 ts_tablet_manager.cc:1905] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:26.122045 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
I20250110 02:51:26.122409 2357 tablet_replica.cc:331] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.122771 2357 raft_consensus.cc:2238] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.123114 2357 raft_consensus.cc:2267] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.123701 2357 ts_tablet_manager.cc:1905] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.125072 1184 tablet_bootstrap.cc:654] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.126140 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:26.125924 2352 ts_tablet_manager.cc:1918] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.126502 1181 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.126786 2352 log.cc:1198] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:26.127269 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:26.127789 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:26.127977 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:26.128121 1181 tablet_bootstrap.cc:654] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.128345 1443 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:26.129434 2357 ts_tablet_manager.cc:1918] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.138001 1188 tablet_bootstrap.cc:492] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.138052 2345 raft_consensus.cc:491] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.138229 1188 ts_tablet_manager.cc:1397] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.001s
I20250110 02:51:26.138219 2345 raft_consensus.cc:513] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.138837 2357 log.cc:1198] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/28340f0f9a2043a199c74f13755eb702
I20250110 02:51:26.139029 2345 leader_election.cc:290] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.139487 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.139732 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.139899 768 raft_consensus.cc:2463] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:26.139918 1188 raft_consensus.cc:357] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.140077 1188 raft_consensus.cc:383] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.140116 1188 raft_consensus.cc:738] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.140338 853 leader_election.cc:304] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:26.140296 1188 consensus_queue.cc:260] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.140772 1188 ts_tablet_manager.cc:1428] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250110 02:51:26.140899 853 leader_election.cc:343] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.141176 2524 raft_consensus.cc:2798] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.141237 2524 raft_consensus.cc:491] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.141295 2524 raft_consensus.cc:3054] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.141495 1181 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.141610 1181 ts_tablet_manager.cc:1397] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250110 02:51:26.142257 1181 raft_consensus.cc:357] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.142441 1181 raft_consensus.cc:383] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.142498 1181 raft_consensus.cc:738] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.142658 1181 consensus_queue.cc:260] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.142366 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:26.143170 1181 ts_tablet_manager.cc:1428] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 02:51:26.143416 1181 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.143605 2524 raft_consensus.cc:513] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.144141 2524 leader_election.cc:290] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.145202 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:26.145982 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.146135 768 raft_consensus.cc:3054] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.148538 768 raft_consensus.cc:2463] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:26.149031 853 leader_election.cc:304] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:26.149353 853 leader_election.cc:343] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.149672 2345 raft_consensus.cc:2798] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.149803 2345 raft_consensus.cc:695] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:26.150146 2345 consensus_queue.cc:237] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.150872 1188 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.152298 1181 tablet_bootstrap.cc:654] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.152781 1188 tablet_bootstrap.cc:654] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.154027 2352 ts_tablet_manager.cc:1939] T dad0d561a8ef498ba13bc813d4cd82fa P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.155131 2224 ts_tablet_manager.cc:1918] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.155522 2357 ts_tablet_manager.cc:1939] T 28340f0f9a2043a199c74f13755eb702 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.156034 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 88) took 1962 ms. Trace:
I20250110 02:51:26.156495 2357 rpcz_store.cc:276] 0110 02:51:24.193191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.201043 (+ 7852us) service_pool.cc:225] Handling call
0110 02:51:26.122343 (+1921300us) ts_tablet_manager.cc:1145] Deleting tablet 28340f0f9a2043a199c74f13755eb702
0110 02:51:26.122351 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.126810 (+ 4459us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.129374 (+ 2564us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.156021 (+ 26647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919825}
I20250110 02:51:26.158107 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 98) took 1945 ms. Trace:
I20250110 02:51:26.158179 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 28340f0f9a2043a199c74f13755eb702 (table default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]) successfully deleted
I20250110 02:51:26.158594 2357 rpcz_store.cc:276] 0110 02:51:24.212562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.213495 (+ 933us) service_pool.cc:225] Handling call
0110 02:51:26.157991 (+1944496us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:26.158001 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.158093 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942842}
W20250110 02:51:26.159857 853 consensus_peers.cc:487] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.160176 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 126) took 1997 ms. Trace:
I20250110 02:51:26.160503 2357 tablet_replica.cc:331] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:26.160565 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:26.160621 2352 rpcz_store.cc:276] 0110 02:51:24.163031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.163063 (+ 32us) service_pool.cc:225] Handling call
0110 02:51:26.034440 (+1871377us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:26.056799 (+ 22359us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.067922 (+ 11123us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.125587 (+ 57665us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.160138 (+ 34551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871242}
I20250110 02:51:26.162441 2224 log.cc:1198] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:26.162482 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet dad0d561a8ef498ba13bc813d4cd82fa (table default.Table_5bfa749055ab45479cf96da68e537ece [id=488d922a6b604812af65ff6b4d9c6606]) successfully deleted
I20250110 02:51:26.162894 2224 ts_tablet_manager.cc:1939] T 2ef72e05f8ed4d46b759c7a04497653b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.164472 1181 tablet_bootstrap.cc:492] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.164778 1181 ts_tablet_manager.cc:1397] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20250110 02:51:26.164503 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 127) took 2000 ms. Trace:
I20250110 02:51:26.165532 1181 raft_consensus.cc:357] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.165151 2352 rpcz_store.cc:276] 0110 02:51:24.163355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.163387 (+ 32us) service_pool.cc:225] Handling call
0110 02:51:26.163057 (+1999670us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:26.163069 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.164205 (+ 1136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999277}
I20250110 02:51:26.166419 1181 raft_consensus.cc:383] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.166525 1181 raft_consensus.cc:738] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:26.166777 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
I20250110 02:51:26.166344 1188 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.166827 1181 consensus_queue.cc:260] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.167065 1188 ts_tablet_manager.cc:1397] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20250110 02:51:26.167588 1181 ts_tablet_manager.cc:1428] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:26.167817 1188 raft_consensus.cc:357] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.168021 1188 raft_consensus.cc:383] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.168079 1188 raft_consensus.cc:738] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.168270 1188 consensus_queue.cc:260] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.168735 1188 ts_tablet_manager.cc:1428] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:26.168982 1188 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.171244 1184 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.171392 1184 ts_tablet_manager.cc:1397] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.001s
I20250110 02:51:26.172003 1184 raft_consensus.cc:357] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.172202 1184 raft_consensus.cc:383] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.172276 1184 raft_consensus.cc:738] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.172490 1184 consensus_queue.cc:260] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.172986 1184 ts_tablet_manager.cc:1428] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:26.173226 1184 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.174738 1184 tablet_bootstrap.cc:654] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.186618 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 109) took 2804 ms. Trace:
I20250110 02:51:26.186724 2224 rpcz_store.cc:276] 0110 02:51:23.381925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.382254 (+ 329us) service_pool.cc:225] Handling call
0110 02:51:26.119764 (+2737510us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:26.119773 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.124603 (+ 4830us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.154278 (+ 29675us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.186594 (+ 32316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728137}
I20250110 02:51:26.187054 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 108) took 2805 ms. Trace:
I20250110 02:51:26.187121 2224 rpcz_store.cc:276] 0110 02:51:23.381819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.382159 (+ 340us) service_pool.cc:225] Handling call
0110 02:51:26.186950 (+2804791us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:26.186958 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.187042 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2794726}
I20250110 02:51:26.187350 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 111) took 2804 ms. Trace:
I20250110 02:51:26.187399 2224 rpcz_store.cc:276] 0110 02:51:23.382640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.383465 (+ 825us) service_pool.cc:225] Handling call
0110 02:51:26.187277 (+2803812us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:26.187284 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.187339 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2794930}
I20250110 02:51:26.187606 2224 tablet_replica.cc:331] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.187769 2224 raft_consensus.cc:2238] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.187937 2224 raft_consensus.cc:2267] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.188503 2224 ts_tablet_manager.cc:1905] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.190587 1181 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.191032 2352 tablet_replica.cc:331] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.192616 1181 tablet_bootstrap.cc:654] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.193454 2357 raft_consensus.cc:2238] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.193655 2357 raft_consensus.cc:2267] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.193694 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2ef72e05f8ed4d46b759c7a04497653b (table default.Table_e298b60e8778429bb52cf546d58fa2d9 [id=62df87bfc41d477fb6044eff0ed5c3ea]) successfully deleted
W20250110 02:51:26.193971 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
W20250110 02:51:26.194239 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:26.194567 2549 raft_consensus.cc:491] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.194686 2549 raft_consensus.cc:513] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.195299 2549 leader_election.cc:290] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.195829 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e015a41a03f84c91a45e46e4b7b5e7a9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.196216 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e015a41a03f84c91a45e46e4b7b5e7a9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.196718 709 leader_election.cc:343] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:26.197201 711 leader_election.cc:343] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.197302 711 leader_election.cc:304] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.197556 2199 raft_consensus.cc:2743] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.197202 2549 raft_consensus.cc:491] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.197950 2549 raft_consensus.cc:513] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.198544 2549 leader_election.cc:290] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.198997 2352 raft_consensus.cc:2238] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.199271 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.199899 2456 raft_consensus.cc:491] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.200007 2456 raft_consensus.cc:513] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:26.200110 709 leader_election.cc:343] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.200527 1188 tablet_bootstrap.cc:654] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.200533 2456 leader_election.cc:290] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.200675 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.201097 711 leader_election.cc:343] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.201128 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.201184 711 leader_election.cc:304] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.201272 913 raft_consensus.cc:2463] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.201545 2549 raft_consensus.cc:2743] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.199287 2352 raft_consensus.cc:2267] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.201793 994 leader_election.cc:304] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.201815 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.201984 768 raft_consensus.cc:2463] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.202006 2456 raft_consensus.cc:2798] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.202080 2456 raft_consensus.cc:491] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.202143 2456 raft_consensus.cc:3054] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.202414 2352 ts_tablet_manager.cc:1905] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.204972 2456 raft_consensus.cc:513] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.205899 2456 leader_election.cc:290] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.205869 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:26.206053 913 raft_consensus.cc:3054] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.207720 2224 ts_tablet_manager.cc:1918] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.207800 2224 log.cc:1198] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:26.208134 2224 ts_tablet_manager.cc:1939] T 60e534bf66f044b88d8cda0f3a1228f4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.208478 913 raft_consensus.cc:2463] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.208827 994 leader_election.cc:304] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.209079 2456 raft_consensus.cc:2798] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.209339 2456 raft_consensus.cc:695] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:26.209563 2456 consensus_queue.cc:237] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.209758 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f5f457d9e4028bfd0b707604c42f5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.209971 768 raft_consensus.cc:3054] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.212582 768 raft_consensus.cc:2463] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.216389 2357 ts_tablet_manager.cc:1905] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.220149 2289 raft_consensus.cc:491] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.220306 2289 raft_consensus.cc:513] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.221027 2289 leader_election.cc:290] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:26.222002 996 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:26.222270 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ce9f6a3bb4e4a8d80ca0b047500a5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.222775 994 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.222867 994 leader_election.cc:304] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.223224 2289 raft_consensus.cc:2743] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.223876 1188 tablet_bootstrap.cc:492] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.224164 1188 ts_tablet_manager.cc:1397] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20250110 02:51:26.224272 2289 raft_consensus.cc:491] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.224385 2289 raft_consensus.cc:513] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.224951 2289 leader_election.cc:290] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.225778 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.226203 994 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:26.226661 996 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:26.226752 996 leader_election.cc:304] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.227044 2289 raft_consensus.cc:2743] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.228790 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 110) took 2846 ms. Trace:
I20250110 02:51:26.229254 1184 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.229450 1184 ts_tablet_manager.cc:1397] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20250110 02:51:26.229506 2224 rpcz_store.cc:276] 0110 02:51:23.382006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.382178 (+ 172us) service_pool.cc:225] Handling call
0110 02:51:26.187555 (+2805377us) ts_tablet_manager.cc:1145] Deleting tablet 60e534bf66f044b88d8cda0f3a1228f4
0110 02:51:26.187562 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.196668 (+ 9106us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.207631 (+ 10963us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.228772 (+ 21141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795140}
I20250110 02:51:26.229971 2549 raft_consensus.cc:491] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.230118 2549 raft_consensus.cc:513] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.230412 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 60e534bf66f044b88d8cda0f3a1228f4 (table default.Table_8df25635dd7e4d58818dc14fc678e221 [id=4893b36e085445c4a454cf47ebe1955b]) successfully deleted
I20250110 02:51:26.230823 2224 tablet_replica.cc:331] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.230808 2549 leader_election.cc:290] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.230707 1184 raft_consensus.cc:357] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.231246 1184 raft_consensus.cc:383] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.231333 1184 raft_consensus.cc:738] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.231494 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.231604 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.231580 1184 consensus_queue.cc:260] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:26.232007 711 leader_election.cc:343] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:26.232113 709 leader_election.cc:343] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.232225 709 leader_election.cc:304] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.232473 2224 raft_consensus.cc:2238] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.232484 1184 ts_tablet_manager.cc:1428] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:26.232733 2549 raft_consensus.cc:2743] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.232753 2224 raft_consensus.cc:2267] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.232990 2345 raft_consensus.cc:491] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.233011 1184 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.233121 2345 raft_consensus.cc:513] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.233250 2224 ts_tablet_manager.cc:1905] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.233355 1188 raft_consensus.cc:357] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.233765 1188 raft_consensus.cc:383] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.233906 1188 raft_consensus.cc:738] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.234252 1188 consensus_queue.cc:260] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.235455 2289 consensus_queue.cc:1035] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.234040 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f97a3be9b49958b1218a275e126d4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.239753 1181 tablet_bootstrap.cc:492] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.244453 1181 ts_tablet_manager.cc:1397] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.004s
I20250110 02:51:26.245201 1181 raft_consensus.cc:357] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.245399 1181 raft_consensus.cc:383] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.245489 1181 raft_consensus.cc:738] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.245767 1181 consensus_queue.cc:260] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.246347 1181 ts_tablet_manager.cc:1428] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:26.246632 1181 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.247370 2357 ts_tablet_manager.cc:1918] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.248118 1181 tablet_bootstrap.cc:654] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.253583 2357 log.cc:1198] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/452223af5f6a4836a0f582d394a9453e
I20250110 02:51:26.254523 2357 ts_tablet_manager.cc:1939] T 452223af5f6a4836a0f582d394a9453e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:26.239923 853 leader_election.cc:343] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 971f97a3be9b49958b1218a275e126d4
I20250110 02:51:26.254770 853 leader_election.cc:304] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.255072 1188 ts_tablet_manager.cc:1428] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250110 02:51:26.255196 2378 raft_consensus.cc:2743] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.234336 2345 leader_election.cc:290] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.255612 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:26.256016 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:26.256492 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:26.257572 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:26.259351 2504 log.cc:826] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Log is configured to *not* fsync() on all Append() calls
I20250110 02:51:26.259660 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:26.259920 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:26.260874 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:26.261117 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:26.261344 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:26.261582 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:26.261775 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:26.262358 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:26.263955 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:26.264503 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:26.267500 2345 raft_consensus.cc:491] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.267619 2345 raft_consensus.cc:513] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.267838 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:26.268173 2345 leader_election.cc:290] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.268818 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:26.269016 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:26.269213 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 3e4d2bb0b992474d8e5d21abaca5411c (table default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde]). Expected version 1 got 0
I20250110 02:51:26.262357 2432 consensus_queue.cc:1035] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.269913 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:26.270083 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:26.270306 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:26.271222 1457 catalog_manager.cc:5526] T 6e758fcd046f4196a3735d4b731544e6 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:26.271909 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 96) took 2068 ms. Trace:
I20250110 02:51:26.267272 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:26.269207 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.271202 2199 raft_consensus.cc:491] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.272920 768 raft_consensus.cc:2388] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:26.272886 2199 raft_consensus.cc:513] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.273360 2199 leader_election.cc:290] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.268955 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e804fe33c8ae489bbc4d3765c64b59bd" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.274089 853 leader_election.cc:304] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.274432 2345 raft_consensus.cc:3054] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.275722 1188 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.276818 2345 raft_consensus.cc:2743] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.277755 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.278216 711 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.280560 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:26.281013 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:26.281287 1188 tablet_bootstrap.cc:654] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.282488 2352 ts_tablet_manager.cc:1918] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.264667 1184 tablet_bootstrap.cc:654] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.282754 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.287657 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
W20250110 02:51:26.288142 709 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.288275 709 leader_election.cc:304] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.272779 2357 rpcz_store.cc:276] 0110 02:51:24.203255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.215059 (+ 11804us) service_pool.cc:225] Handling call
0110 02:51:26.160463 (+1945404us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:26.160471 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.244402 (+ 83931us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.247277 (+ 2875us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.271880 (+ 24603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945280}
I20250110 02:51:26.288599 2199 raft_consensus.cc:2743] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.288909 2352 log.cc:1198] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8f4778347f054b86af697dbe3b287263
I20250110 02:51:26.288623 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:26.289320 2357 tablet_replica.cc:331] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.289004 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 452223af5f6a4836a0f582d394a9453e (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]) successfully deleted
I20250110 02:51:26.289544 2357 raft_consensus.cc:2238] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.289773 2357 pending_rounds.cc:62] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:26.289844 2357 pending_rounds.cc:69] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611780677496832 op_type: NO_OP noop_request { }
I20250110 02:51:26.289987 2357 raft_consensus.cc:2883] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:26.289997 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:26.290105 2357 raft_consensus.cc:2267] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.290366 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:26.291113 2357 ts_tablet_manager.cc:1905] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.291534 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:26.291464 2352 ts_tablet_manager.cc:1939] T 8f4778347f054b86af697dbe3b287263 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.291844 2224 ts_tablet_manager.cc:1918] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.291877 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:26.291965 2224 log.cc:1198] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:26.292309 2224 ts_tablet_manager.cc:1939] T bcc9a4a842914a6f878d01194cd9a5f7 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.292302 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:26.292446 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:26.292747 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:26.293382 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:26.295986 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 128) took 2132 ms. Trace:
I20250110 02:51:26.297474 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:26.296712 2352 rpcz_store.cc:276] 0110 02:51:24.163585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.163615 (+ 30us) service_pool.cc:225] Handling call
0110 02:51:26.190985 (+2027370us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:26.190994 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.210633 (+ 19639us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.282126 (+ 71493us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.295812 (+ 13686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027253}
I20250110 02:51:26.298794 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8f4778347f054b86af697dbe3b287263 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]) successfully deleted
I20250110 02:51:26.301937 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
W20250110 02:51:26.302428 1441 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b937188bf2949af8f54bade4e0ad984 consensus state has no leader
I20250110 02:51:26.302569 1441 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 100) took 2022 ms. Trace:
I20250110 02:51:26.302651 1441 rpcz_store.cc:276] 0110 02:51:24.280259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.320425 (+1040166us) service_pool.cc:225] Handling call
0110 02:51:25.331970 (+ 11545us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:25.332188 (+ 218us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:25.332362 (+ 174us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:25.332442 (+ 80us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:25.402264 (+ 69822us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:26.302533 (+900269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.362484 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.362642 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.362652 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:25.362656 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:25.362796 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.362813 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.362820 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.362902 (+ 82us) tablet.cc:698] Row locks acquired
0110 02:51:25.362906 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:25.362949 (+ 43us) write_op.cc:270] Start()
0110 02:51:25.362989 (+ 40us) write_op.cc:276] Timestamp: P: 1736477485362909 usec, L: 0
0110 02:51:25.362995 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.363238 (+ 243us) log.cc:844] Serialized 405 byte log entry
0110 02:51:25.365909 (+ 2671us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.366019 (+ 110us) write_op.cc:301] APPLY: starting
0110 02:51:25.366031 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.366070 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.366104 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.366147 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.366194 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.366201 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:25.366261 (+ 60us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.366314 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.366328 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:25.366333 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.366418 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.366421 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":375,"apply.run_wall_time_us":374,"num_ops":1,"prepare.queue_time_us":29528,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":880,"raft.queue_time_us":37,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":224,"replication_time_us":2887}]]}
I20250110 02:51:26.304056 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 129) took 2140 ms. Trace:
I20250110 02:51:26.304515 2352 rpcz_store.cc:276] 0110 02:51:24.163799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.163827 (+ 28us) service_pool.cc:225] Handling call
0110 02:51:26.299938 (+2136111us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:26.299949 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.304039 (+ 4090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135729}
W20250110 02:51:26.305541 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
I20250110 02:51:26.306622 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 130) took 2142 ms. Trace:
I20250110 02:51:26.307320 2600 raft_consensus.cc:491] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.308692 1448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d] alter complete (version 6)
I20250110 02:51:26.311435 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.311465 880 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.311643 2199 raft_consensus.cc:491] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.311756 2199 raft_consensus.cc:513] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.312290 2199 leader_election.cc:290] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:26.312469 1448 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1edb4e06e40c43438882e1eb622d788e consensus state has no leader
I20250110 02:51:26.312829 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.313331 709 leader_election.cc:343] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.314605 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.314785 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.314941 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a4745356e3423f9d4892d242da1c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315070 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315213 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315344 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315479 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315635 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315763 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.315886 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316031 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316151 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316318 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316447 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316592 883 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316731 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316797 896 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316870 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.316972 898 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317113 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317260 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317267 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.312812 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.317402 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317523 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317636 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:26.317699 711 leader_election.cc:343] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.317759 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317791 711 leader_election.cc:304] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.317888 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.318186 2199 raft_consensus.cc:2743] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.318365 893 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.318456 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.318827 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.319186 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.319576 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.320240 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.320544 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.320919 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.306833 2352 rpcz_store.cc:276] 0110 02:51:24.164000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.164029 (+ 29us) service_pool.cc:225] Handling call
0110 02:51:26.306071 (+2142042us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:26.306081 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.306457 (+ 376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2141932}
I20250110 02:51:26.321593 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 131) took 2157 ms. Trace:
W20250110 02:51:26.321713 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 452223af5f6a4836a0f582d394a9453e already in progress: creating tablet
I20250110 02:51:26.314275 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 114) took 2930 ms. Trace:
I20250110 02:51:26.321720 2352 rpcz_store.cc:276] 0110 02:51:24.164230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.164266 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:26.321507 (+2157241us) ts_tablet_manager.cc:1145] Deleting tablet 08705d260c0a462b94ba6b0dc5121123
0110 02:51:26.321515 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.321581 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157126}
I20250110 02:51:26.321995 2224 rpcz_store.cc:276] 0110 02:51:23.383987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.385612 (+ 1625us) service_pool.cc:225] Handling call
0110 02:51:26.230784 (+2845172us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:26.230792 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.242473 (+ 11681us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.291748 (+ 49275us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.314244 (+ 22496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2835030}
W20250110 02:51:26.322438 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08705d260c0a462b94ba6b0dc5121123 already in progress: creating tablet
I20250110 02:51:26.318979 2478 consensus_queue.cc:1035] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250110 02:51:26.323191 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.323204 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.323660 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 019c0d60da5548bb86c93ccd89bf94ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.324250 881 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.324621 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.324959 881 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.325340 881 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.325697 881 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.326040 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.326913 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:26.327126 996 consensus_peers.cc:487] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 747ae8cc8c2c4ffa9071e23f7d7c5e11. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.327940 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.328187 1181 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.328333 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.328646 1181 ts_tablet_manager.cc:1397] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.000s
I20250110 02:51:26.328702 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.329035 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.329349 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.329416 1181 raft_consensus.cc:357] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.329591 1181 raft_consensus.cc:383] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.329664 1181 raft_consensus.cc:738] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.329857 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.330302 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.317375 2600 raft_consensus.cc:513] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.330497 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.330775 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.331104 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.331286 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d970e2fe72b4727968ca605b3e772cc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.331311 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet bcc9a4a842914a6f878d01194cd9a5f7 (table default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]) successfully deleted
I20250110 02:51:26.331426 2224 tablet_replica.cc:331] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.331425 917 raft_consensus.cc:2463] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.331606 2224 raft_consensus.cc:2238] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.331743 2224 raft_consensus.cc:2267] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.331715 994 leader_election.cc:304] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.331966 2478 raft_consensus.cc:2798] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.332044 2478 raft_consensus.cc:491] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.332108 2478 raft_consensus.cc:3054] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:26.334713 1448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 106) took 3019 ms (client timeout 1000 ms). Trace:
I20250110 02:51:26.335150 2478 raft_consensus.cc:513] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:26.334836 1448 rpcz_store.cc:269] 0110 02:51:23.315756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.351031 (+ 35275us) service_pool.cc:225] Handling call
0110 02:51:23.954913 (+603882us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.954919 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:25.038539 (+1083620us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.038548 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:26.334680 (+1296132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:23.880846 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.880951 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.880958 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:23.880961 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:23.881059 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.881072 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.881077 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:23.881143 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:23.881146 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:23.881174 (+ 28us) write_op.cc:270] Start()
0110 02:51:23.881203 (+ 29us) write_op.cc:276] Timestamp: P: 1736477483881132 usec, L: 0
0110 02:51:23.881207 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.881365 (+ 158us) log.cc:844] Serialized 527 byte log entry
0110 02:51:23.889109 (+ 7744us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.889280 (+ 171us) write_op.cc:301] APPLY: starting
0110 02:51:23.889300 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.889349 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.889387 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.889458 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.889499 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:23.889509 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:23.889594 (+ 85us) log.cc:844] Serialized 23 byte log entry
0110 02:51:23.889683 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.889710 (+ 27us) write_op.cc:454] Released schema lock
0110 02:51:23.889719 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:23.889812 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.889818 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:23.949723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:23.949839 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:23.949856 (+ 17us) write_op.cc:435] Acquired schema lock
0110 02:51:23.949858 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:23.949966 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:23.949980 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:23.949984 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:23.950047 (+ 63us) tablet.cc:698] Row locks acquired
0110 02:51:23.950049 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:23.950083 (+ 34us) write_op.cc:270] Start()
0110 02:51:23.950113 (+ 30us) write_op.cc:276] Timestamp: P: 1736477483950033 usec, L: 0
0110 02:51:23.950116 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:23.950270 (+ 154us) log.cc:844] Serialized 356 byte log entry
0110 02:51:23.953166 (+ 2896us) op_driver.cc:464] REPLICATION: finished
0110 02:51:23.953299 (+ 133us) write_op.cc:301] APPLY: starting
0110 02:51:23.953314 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:23.953349 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:23.953390 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:23.953439 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:23.953470 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:23.953481 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:23.953546 (+ 65us) log.cc:844] Serialized 23 byte log entry
0110 02:51:23.953600 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:23.953616 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:23.953622 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:25.022274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.022369 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.022376 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.022379 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.022547 (+ 168us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.022560 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.022564 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.022622 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:25.022624 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.022656 (+ 32us) write_op.cc:270] Start()
0110 02:51:25.022685 (+ 29us) write_op.cc:276] Timestamp: P: 1736477485022601 usec, L: 0
0110 02:51:25.022689 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.022878 (+ 189us) log.cc:844] Serialized 356 byte log entry
0110 02:51:25.026873 (+ 3995us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.027078 (+ 205us) write_op.cc:301] APPLY: starting
0110 02:51:25.027093 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.027132 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.027185 (+ 53us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.027232 (+ 47us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.027260 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.027269 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:25.027332 (+ 63us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.027385 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.027397 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:25.027402 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":11768,"reported_tablets":280,"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":519,"num_ops":1,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":631,"prepare.run_wall_time_us":631,"raft.queue_time_us":143,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":223,"replication_time_us":7866}],["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":370,"num_ops":1,"prepare.queue_time_us":11477,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":643,"raft.queue_time_us":804,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":274,"replication_time_us":3026}],["op",{"apply.queue_time_us":161,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":372,"num_ops":1,"prepare.queue_time_us":17982,"prepare.run_cpu_time_us":679,"prepare.run_wall_time_us":725,"raft.queue_time_us":1441,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":255,"replication_time_us":4159}]]}
I20250110 02:51:26.336127 2478 leader_election.cc:290] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.336597 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d970e2fe72b4727968ca605b3e772cc" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:26.336714 2600 leader_election.cc:290] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.336763 917 raft_consensus.cc:3054] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.337766 1181 consensus_queue.cc:260] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.338274 1181 ts_tablet_manager.cc:1428] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250110 02:51:26.338552 1181 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.339454 917 raft_consensus.cc:2463] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.339931 994 leader_election.cc:304] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:26.340070 1181 tablet_bootstrap.cc:654] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.341190 886 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.341564 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.341953 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.342322 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.342633 886 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.344239 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.344902 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a with delete_type TABLET_DATA_DELETED (replaced by tablet dc6b2654f86e4686b8a3034de02d11bf) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:26.322667 2352 tablet_replica.cc:331] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:26.345458 996 leader_election.cc:343] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:26.346407 2600 raft_consensus.cc:2798] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.346314 2352 raft_consensus.cc:2238] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:26.346747 996 leader_election.cc:343] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:26.347203 2478 consensus_queue.cc:1035] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.347111 2352 raft_consensus.cc:2267] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:26.348256 853 consensus_peers.cc:487] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.348868 2600 raft_consensus.cc:695] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:26.349191 2600 consensus_queue.cc:237] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.349419 2224 ts_tablet_manager.cc:1905] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.350199 2345 raft_consensus.cc:491] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.350342 2345 raft_consensus.cc:513] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.350461 2352 ts_tablet_manager.cc:1905] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.350966 2345 leader_election.cc:290] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.351446 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.351507 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.351619 1061 raft_consensus.cc:2463] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:26.351683 768 raft_consensus.cc:2463] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:26.351964 853 leader_election.cc:304] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:26.352172 2345 raft_consensus.cc:2798] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.352237 2345 raft_consensus.cc:491] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.352288 2345 raft_consensus.cc:3054] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.354687 2345 raft_consensus.cc:513] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.355234 2345 leader_election.cc:290] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.358687 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.358857 768 raft_consensus.cc:3054] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.361178 768 raft_consensus.cc:2463] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:26.361654 853 leader_election.cc:304] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:26.362471 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d424b4da66184bd19eccefad80621516" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:26.362651 1061 raft_consensus.cc:3054] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.365384 1061 raft_consensus.cc:2463] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:26.365947 2345 raft_consensus.cc:2798] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.366061 2345 raft_consensus.cc:695] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:26.366263 2345 consensus_queue.cc:237] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.369302 1184 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.372275 1184 ts_tablet_manager.cc:1397] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.001s
I20250110 02:51:26.373234 1184 raft_consensus.cc:357] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.374302 1184 raft_consensus.cc:383] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.378455 1184 raft_consensus.cc:738] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.378737 1184 consensus_queue.cc:260] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.379424 1184 ts_tablet_manager.cc:1428] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:26.379716 1184 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.381417 1184 tablet_bootstrap.cc:654] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.386866 917 raft_consensus.cc:3054] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.389989 2352 ts_tablet_manager.cc:1918] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.390817 2352 log.cc:1198] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/a8cbed1ba6f04fd9a43a8cd6b7adc7b8
I20250110 02:51:26.393141 2352 ts_tablet_manager.cc:1939] T a8cbed1ba6f04fd9a43a8cd6b7adc7b8 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.397518 2345 raft_consensus.cc:491] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.397647 1181 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.397646 2345 raft_consensus.cc:513] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.398690 1181 ts_tablet_manager.cc:1397] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
W20250110 02:51:26.399415 853 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:26.399550 2345 leader_election.cc:290] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.399772 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3510e86314144a2ca012aacf3d78eacc" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.399966 1061 raft_consensus.cc:2463] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:26.400270 853 leader_election.cc:304] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.400476 2378 raft_consensus.cc:2798] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.400543 2378 raft_consensus.cc:491] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.400606 2378 raft_consensus.cc:3054] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.401614 2224 ts_tablet_manager.cc:1918] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.401717 2224 log.cc:1198] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/2e25f9643ac0476a90c93fb9aeb921a3
I20250110 02:51:26.401968 2224 ts_tablet_manager.cc:1939] T 2e25f9643ac0476a90c93fb9aeb921a3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.402685 2378 raft_consensus.cc:513] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.403270 2378 leader_election.cc:290] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.404139 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3510e86314144a2ca012aacf3d78eacc" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:26.404387 1061 raft_consensus.cc:3054] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:26.406827 853 leader_election.cc:343] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:26.407996 1061 raft_consensus.cc:2463] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:26.408450 853 leader_election.cc:304] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.408985 2378 raft_consensus.cc:2798] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.409168 2378 raft_consensus.cc:695] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:26.409453 2378 consensus_queue.cc:237] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.413192 1188 tablet_bootstrap.cc:492] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.413344 1188 ts_tablet_manager.cc:1397] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.138s user 0.006s sys 0.008s
I20250110 02:51:26.413479 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 112) took 3030 ms. Trace:
I20250110 02:51:26.414073 1188 raft_consensus.cc:357] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.407438 1181 raft_consensus.cc:357] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.414240 1188 raft_consensus.cc:383] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.414299 1188 raft_consensus.cc:738] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.414315 1181 raft_consensus.cc:383] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.414523 1181 raft_consensus.cc:738] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.414498 1188 consensus_queue.cc:260] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.414965 1188 ts_tablet_manager.cc:1428] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.414172 2224 rpcz_store.cc:276] 0110 02:51:23.382734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.383446 (+ 712us) service_pool.cc:225] Handling call
0110 02:51:26.330475 (+2947029us) ts_tablet_manager.cc:1145] Deleting tablet 2e25f9643ac0476a90c93fb9aeb921a3
0110 02:51:26.331369 (+ 894us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.359784 (+ 28415us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.401514 (+ 41730us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.413448 (+ 11934us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5559,"tablet-delete.queue_time_us":2934640}
I20250110 02:51:26.415616 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 2e25f9643ac0476a90c93fb9aeb921a3 (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]) successfully deleted
I20250110 02:51:26.420114 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 132) took 2255 ms. Trace:
I20250110 02:51:26.420238 2352 rpcz_store.cc:276] 0110 02:51:24.164483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.164701 (+ 218us) service_pool.cc:225] Handling call
0110 02:51:26.322413 (+2157712us) ts_tablet_manager.cc:1145] Deleting tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8
0110 02:51:26.322420 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.370240 (+ 47820us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.389411 (+ 19171us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.420094 (+ 30683us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157555}
I20250110 02:51:26.420799 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 (table default.Table_e55e81b07bfe4a668255066cb6447735 [id=f7fcd16a9d4944e0a4ce181abad7237d]) successfully deleted
I20250110 02:51:26.421602 2352 tablet_replica.cc:331] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.421825 2352 raft_consensus.cc:2238] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.422013 2352 raft_consensus.cc:2267] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.422798 2352 ts_tablet_manager.cc:1905] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.423630 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:26.424558 2224 tablet_replica.cc:331] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.425980 2224 raft_consensus.cc:2238] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.427057 1188 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.415198 1181 consensus_queue.cc:260] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.431735 1181 ts_tablet_manager.cc:1428] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250110 02:51:26.432024 2224 raft_consensus.cc:2267] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.436959 2224 ts_tablet_manager.cc:1905] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.443331 1181 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.444147 2199 raft_consensus.cc:491] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.444275 2199 raft_consensus.cc:513] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.444828 2199 leader_election.cc:290] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.445479 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.445917 711 leader_election.cc:343] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.450668 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.451130 709 leader_election.cc:343] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.451223 709 leader_election.cc:304] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.451535 2199 raft_consensus.cc:2743] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.453531 2224 ts_tablet_manager.cc:1918] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.461823 1188 tablet_bootstrap.cc:654] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.462044 2345 raft_consensus.cc:491] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.463191 2345 raft_consensus.cc:513] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.464465 1184 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
W20250110 02:51:26.464704 853 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:26.465020 1184 ts_tablet_manager.cc:1397] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.002s
I20250110 02:51:26.465292 2345 leader_election.cc:290] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.465307 2199 raft_consensus.cc:491] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.465499 2199 raft_consensus.cc:513] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.466015 2199 leader_election.cc:290] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.466486 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.457566 1181 tablet_bootstrap.cc:654] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.463425 2478 raft_consensus.cc:491] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:26.466930 709 leader_election.cc:343] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.467021 2478 raft_consensus.cc:513] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.467645 2478 leader_election.cc:290] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.467902 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452223af5f6a4836a0f582d394a9453e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.468343 994 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.465480 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302af84fd8374ffa96a7503b54849536" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.466710 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.468473 996 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
I20250110 02:51:26.468580 996 leader_election.cc:304] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.468791 1061 raft_consensus.cc:2463] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:26.468983 2478 raft_consensus.cc:2743] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:26.468979 711 leader_election.cc:343] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.469079 711 leader_election.cc:304] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.469168 853 leader_election.cc:304] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.469573 2345 raft_consensus.cc:2798] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.469684 2345 raft_consensus.cc:491] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.469708 2199 raft_consensus.cc:2743] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.469771 2345 raft_consensus.cc:3054] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.466892 2224 log.cc:1198] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:26.471958 2224 ts_tablet_manager.cc:1939] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.475617 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 120) took 3084 ms. Trace:
I20250110 02:51:26.478868 2345 raft_consensus.cc:513] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.479133 2224 rpcz_store.cc:276] 0110 02:51:23.391107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.391255 (+ 148us) service_pool.cc:225] Handling call
0110 02:51:26.424497 (+3033242us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:26.424508 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.449349 (+ 24841us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.453395 (+ 4046us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.475584 (+ 22189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3023585}
I20250110 02:51:26.479686 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]) successfully deleted
I20250110 02:51:26.480116 2224 tablet_replica.cc:331] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.480368 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:26.481348 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:26.481952 2224 raft_consensus.cc:2238] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.481947 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:26.482156 2224 raft_consensus.cc:2267] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.482658 2224 ts_tablet_manager.cc:1905] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.482542 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
W20250110 02:51:26.483752 853 leader_election.cc:343] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:26.483608 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:26.482949 1184 raft_consensus.cc:357] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.484156 2352 ts_tablet_manager.cc:1918] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.484256 2352 log.cc:1198] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:26.484398 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302af84fd8374ffa96a7503b54849536" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:26.484557 2352 ts_tablet_manager.cc:1939] T a7397e41f7374e9fb1621dbcf30af703 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.484581 1061 raft_consensus.cc:3054] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.484838 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 133) took 2319 ms. Trace:
I20250110 02:51:26.484578 1184 raft_consensus.cc:383] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.484935 2352 rpcz_store.cc:276] 0110 02:51:24.165020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.165061 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:26.421084 (+2256023us) ts_tablet_manager.cc:1145] Deleting tablet a7397e41f7374e9fb1621dbcf30af703
0110 02:51:26.421559 (+ 475us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.425723 (+ 4164us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.484054 (+ 58331us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.484802 (+ 748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255892}
I20250110 02:51:26.484998 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:26.484963 1184 raft_consensus.cc:738] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.485321 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet a7397e41f7374e9fb1621dbcf30af703 (table default.Table_94cf6a20190647dd87d3616b069b0243 [id=a91e19e65fd74454b4b4d339a5cbe806]) successfully deleted
I20250110 02:51:26.485632 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:26.486402 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:26.486326 1184 consensus_queue.cc:260] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.487483 1061 raft_consensus.cc:2463] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:26.487879 853 leader_election.cc:304] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.488206 2377 raft_consensus.cc:2798] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.488333 2377 raft_consensus.cc:695] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:26.488456 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:26.488581 2377 consensus_queue.cc:237] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.489131 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:26.489923 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:26.490497 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:26.492511 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:26.493947 1188 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.494132 1188 ts_tablet_manager.cc:1397] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.000s
I20250110 02:51:26.494279 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:26.495345 1188 raft_consensus.cc:357] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.495563 1188 raft_consensus.cc:383] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.495666 1188 raft_consensus.cc:738] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.495954 1188 consensus_queue.cc:260] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.496584 1188 ts_tablet_manager.cc:1428] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250110 02:51:26.496793 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9b937188bf2949af8f54bade4e0ad984 consensus state has no leader
I20250110 02:51:26.496945 1188 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.497079 1184 ts_tablet_manager.cc:1428] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250110 02:51:26.497439 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:26.499621 1184 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.499895 2345 leader_election.cc:290] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.501224 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:26.504657 2357 ts_tablet_manager.cc:1918] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.504801 2357 log.cc:1198] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:26.505172 2357 ts_tablet_manager.cc:1939] T 8eb9ead3175140028e584e1ccb4874de P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.505486 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 99) took 2292 ms. Trace:
I20250110 02:51:26.505594 2357 rpcz_store.cc:276] 0110 02:51:24.212639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.213621 (+ 982us) service_pool.cc:225] Handling call
0110 02:51:26.289193 (+2075572us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:26.289205 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.396535 (+107330us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.504557 (+108022us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.505472 (+ 915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2073990}
I20250110 02:51:26.506050 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8eb9ead3175140028e584e1ccb4874de (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:26.506359 2357 tablet_replica.cc:331] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.506592 2357 raft_consensus.cc:2238] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.506793 2357 raft_consensus.cc:2267] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.507319 2357 ts_tablet_manager.cc:1905] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.510208 886 tablet_service.cc:1467] Processing CreateTablet for tablet 379f6261031c4928a364d5da7e3d55ea (DEFAULT_TABLE table=default.Table_150713d6bc354b23b78ed7f02cb4da55 [id=2ea773ba21984732b7d7283dbbb7b675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.511256 1181 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.511466 749 tablet_service.cc:1467] Processing CreateTablet for tablet bc7f93f016e949de8f7b1dcaa150480c (DEFAULT_TABLE table=default.Table_7cf301ad33854cc0b24ddec01dbdbdd1 [id=179f019206774adfa60a045455600ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.511529 1181 ts_tablet_manager.cc:1397] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.004s
I20250110 02:51:26.511775 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7f93f016e949de8f7b1dcaa150480c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.513180 1181 raft_consensus.cc:357] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.513469 1181 raft_consensus.cc:383] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.513553 1181 raft_consensus.cc:738] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.513914 1181 consensus_queue.cc:260] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.514578 751 tablet_service.cc:1467] Processing CreateTablet for tablet 379f6261031c4928a364d5da7e3d55ea (DEFAULT_TABLE table=default.Table_150713d6bc354b23b78ed7f02cb4da55 [id=2ea773ba21984732b7d7283dbbb7b675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.514859 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f6261031c4928a364d5da7e3d55ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.516135 1181 ts_tablet_manager.cc:1428] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250110 02:51:26.516745 750 tablet_service.cc:1467] Processing CreateTablet for tablet e848586bec574a529caaf5e498db79cd (DEFAULT_TABLE table=default.Table_01904216609c46c581a0b70981481bb9 [id=828a0eaadd114fb1a1e6cd147125dd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.517002 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e848586bec574a529caaf5e498db79cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.517756 2199 raft_consensus.cc:491] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.517887 2199 raft_consensus.cc:513] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.518627 2199 leader_election.cc:290] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.519846 1181 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.521577 881 tablet_service.cc:1467] Processing CreateTablet for tablet bc7f93f016e949de8f7b1dcaa150480c (DEFAULT_TABLE table=default.Table_7cf301ad33854cc0b24ddec01dbdbdd1 [id=179f019206774adfa60a045455600ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.521608 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.521947 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7f93f016e949de8f7b1dcaa150480c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.522022 893 tablet_service.cc:1467] Processing CreateTablet for tablet e848586bec574a529caaf5e498db79cd (DEFAULT_TABLE table=default.Table_01904216609c46c581a0b70981481bb9 [id=828a0eaadd114fb1a1e6cd147125dd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.522337 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e848586bec574a529caaf5e498db79cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.522748 2224 ts_tablet_manager.cc:1918] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.522847 2224 log.cc:1198] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:26.523159 2224 ts_tablet_manager.cc:1939] T 7d4422eca5604d149e73c2a669d07af0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.523186 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:26.523415 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 118) took 3137 ms. Trace:
I20250110 02:51:26.523782 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:26.523979 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:26.524302 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:26.524538 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:26.525301 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:26.526790 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 379f6261031c4928a364d5da7e3d55ea (DEFAULT_TABLE table=default.Table_150713d6bc354b23b78ed7f02cb4da55 [id=2ea773ba21984732b7d7283dbbb7b675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.527033 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e848586bec574a529caaf5e498db79cd (DEFAULT_TABLE table=default.Table_01904216609c46c581a0b70981481bb9 [id=828a0eaadd114fb1a1e6cd147125dd7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.527156 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f6261031c4928a364d5da7e3d55ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.527328 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e848586bec574a529caaf5e498db79cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.527617 1024 tablet_service.cc:1467] Processing CreateTablet for tablet bc7f93f016e949de8f7b1dcaa150480c (DEFAULT_TABLE table=default.Table_7cf301ad33854cc0b24ddec01dbdbdd1 [id=179f019206774adfa60a045455600ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.527884 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7f93f016e949de8f7b1dcaa150480c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.528030 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f6261031c4928a364d5da7e3d55ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.533033 889 tablet_service.cc:1467] Processing CreateTablet for tablet 91f268832c914b88958d6336aa8813af (DEFAULT_TABLE table=default.Table_94cd6ab467f442cba51ee627721ab3e8 [id=f59f8a43bfbe4e99a99f2ebd1ac77ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.533401 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f268832c914b88958d6336aa8813af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.533620 881 tablet_service.cc:1467] Processing CreateTablet for tablet b5f162de87044005bdafe93292ff71b8 (DEFAULT_TABLE table=default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.534056 1038 tablet_service.cc:1467] Processing CreateTablet for tablet b5f162de87044005bdafe93292ff71b8 (DEFAULT_TABLE table=default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.534432 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 91f268832c914b88958d6336aa8813af (DEFAULT_TABLE table=default.Table_94cd6ab467f442cba51ee627721ab3e8 [id=f59f8a43bfbe4e99a99f2ebd1ac77ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.534554 1022 tablet_service.cc:1467] Processing CreateTablet for tablet e23dff2b3a99464798847aa35dc5f584 (DEFAULT_TABLE table=default.Table_a3b9a4a35b4243988af7de8340b26084 [id=08fa7ecbdd324648a5b3a972b6bf02fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.534731 1039 tablet_service.cc:1467] Processing CreateTablet for tablet e0a721324b6a4ffc92bc2f73adccf0e0 (DEFAULT_TABLE table=default.Table_1929f7bc73fd49cda28f6dc68e8983f1 [id=96466ef113bb4d43897c315b7735b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.534927 880 tablet_service.cc:1467] Processing CreateTablet for tablet c8825527c0cf4b889a4f7f5e5faa2f01 (DEFAULT_TABLE table=default.Table_60139b9c4e354551888ab15aa5329832 [id=871c941f0b334092affc2c3c22deaab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.535008 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f162de87044005bdafe93292ff71b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.535238 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8825527c0cf4b889a4f7f5e5faa2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.521286 1184 tablet_bootstrap.cc:654] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.531155 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1d064a7e8ea34f18b42407903daf7197 (DEFAULT_TABLE table=default.Table_7d36d6648c3f4c45b1672434450cc64b [id=89aaf73e8d76412c8e91bbacdcaefb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.538589 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2d9c4bce3419a9d8f6f9808549e08 (DEFAULT_TABLE table=default.Table_f38a47fe3eca45f2ab04c52b70cc6c79 [id=e434762c7a1e4e4580bed8e5ff3944ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.538661 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d064a7e8ea34f18b42407903daf7197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.538625 883 tablet_service.cc:1467] Processing CreateTablet for tablet e23dff2b3a99464798847aa35dc5f584 (DEFAULT_TABLE table=default.Table_a3b9a4a35b4243988af7de8340b26084 [id=08fa7ecbdd324648a5b3a972b6bf02fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.538899 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2d9c4bce3419a9d8f6f9808549e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.539304 2549 raft_consensus.cc:491] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.539404 2549 raft_consensus.cc:513] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.540030 2549 leader_election.cc:290] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.534554 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1d064a7e8ea34f18b42407903daf7197 (DEFAULT_TABLE table=default.Table_7d36d6648c3f4c45b1672434450cc64b [id=89aaf73e8d76412c8e91bbacdcaefb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.540519 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.531224 747 tablet_service.cc:1467] Processing CreateTablet for tablet e0a721324b6a4ffc92bc2f73adccf0e0 (DEFAULT_TABLE table=default.Table_1929f7bc73fd49cda28f6dc68e8983f1 [id=96466ef113bb4d43897c315b7735b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.542573 895 tablet_service.cc:1467] Processing CreateTablet for tablet 1d064a7e8ea34f18b42407903daf7197 (DEFAULT_TABLE table=default.Table_7d36d6648c3f4c45b1672434450cc64b [id=89aaf73e8d76412c8e91bbacdcaefb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.542641 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a721324b6a4ffc92bc2f73adccf0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.542903 894 tablet_service.cc:1467] Processing CreateTablet for tablet e0a721324b6a4ffc92bc2f73adccf0e0 (DEFAULT_TABLE table=default.Table_1929f7bc73fd49cda28f6dc68e8983f1 [id=96466ef113bb4d43897c315b7735b361]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.543207 898 tablet_service.cc:1467] Processing CreateTablet for tablet 471862cae972410b9770f7a8742cac0a (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.543484 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:26.543568 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2d9c4bce3419a9d8f6f9808549e08 (DEFAULT_TABLE table=default.Table_f38a47fe3eca45f2ab04c52b70cc6c79 [id=e434762c7a1e4e4580bed8e5ff3944ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.543927 1031 tablet_service.cc:1467] Processing CreateTablet for tablet c8825527c0cf4b889a4f7f5e5faa2f01 (DEFAULT_TABLE table=default.Table_60139b9c4e354551888ab15aa5329832 [id=871c941f0b334092affc2c3c22deaab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.544224 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 471862cae972410b9770f7a8742cac0a (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.544584 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d064a7e8ea34f18b42407903daf7197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.531301 2199 raft_consensus.cc:491] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.531436 755 tablet_service.cc:1467] Processing CreateTablet for tablet 471862cae972410b9770f7a8742cac0a (DEFAULT_TABLE table=default.Table_dab0925165d74fb087cc0f47cca63c92 [id=16e264ced81d48f496f1c4738911f73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.546581 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471862cae972410b9770f7a8742cac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.547703 751 tablet_service.cc:1467] Processing CreateTablet for tablet b5f162de87044005bdafe93292ff71b8 (DEFAULT_TABLE table=default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.548035 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f162de87044005bdafe93292ff71b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.548657 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a721324b6a4ffc92bc2f73adccf0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.531436 744 tablet_service.cc:1467] Processing CreateTablet for tablet 91f268832c914b88958d6336aa8813af (DEFAULT_TABLE table=default.Table_94cd6ab467f442cba51ee627721ab3e8 [id=f59f8a43bfbe4e99a99f2ebd1ac77ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.550597 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f268832c914b88958d6336aa8813af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.553392 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471862cae972410b9770f7a8742cac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.531610 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd2d9c4bce3419a9d8f6f9808549e08 (DEFAULT_TABLE table=default.Table_f38a47fe3eca45f2ab04c52b70cc6c79 [id=e434762c7a1e4e4580bed8e5ff3944ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.554597 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2d9c4bce3419a9d8f6f9808549e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.532066 752 tablet_service.cc:1467] Processing CreateTablet for tablet c8825527c0cf4b889a4f7f5e5faa2f01 (DEFAULT_TABLE table=default.Table_60139b9c4e354551888ab15aa5329832 [id=871c941f0b334092affc2c3c22deaab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.558418 2289 raft_consensus.cc:491] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.558624 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8825527c0cf4b889a4f7f5e5faa2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.558547 2289 raft_consensus.cc:513] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.559218 2289 leader_election.cc:290] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.559775 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.560221 994 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.532270 741 tablet_service.cc:1467] Processing CreateTablet for tablet e23dff2b3a99464798847aa35dc5f584 (DEFAULT_TABLE table=default.Table_a3b9a4a35b4243988af7de8340b26084 [id=08fa7ecbdd324648a5b3a972b6bf02fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:26.562606 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23dff2b3a99464798847aa35dc5f584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.565054 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23dff2b3a99464798847aa35dc5f584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.565220 2345 consensus_queue.cc:1035] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:26.532513 709 leader_election.cc:343] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:26.566537 711 leader_election.cc:343] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.566747 711 leader_election.cc:304] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.567135 2199 raft_consensus.cc:513] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:26.567420 711 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.567588 2549 raft_consensus.cc:2743] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.567649 2199 leader_election.cc:290] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.567692 2524 consensus_queue.cc:1035] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.567714 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:26.567855 764 raft_consensus.cc:2463] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:26.568130 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00e97e496b54dcf82b7182591d6f4aa" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.568585 711 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.568565 996 leader_election.cc:304] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:26.568311 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:26.569566 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:26.570153 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:26.570546 2289 raft_consensus.cc:2798] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:26.570629 2289 raft_consensus.cc:491] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:26.570695 2289 raft_consensus.cc:3054] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.571272 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:26.573026 2289 raft_consensus.cc:513] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.573909 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:26.574038 764 raft_consensus.cc:3054] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.574216 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:26.574646 994 leader_election.cc:343] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.576042 2289 leader_election.cc:290] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.576241 764 raft_consensus.cc:2463] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:26.576591 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00e97e496b54dcf82b7182591d6f4aa" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.576833 996 leader_election.cc:304] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:26.576930 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.577375 709 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.577466 709 leader_election.cc:304] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:26.577795 709 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.577872 709 leader_election.cc:304] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.578073 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:26.578239 2549 raft_consensus.cc:2743] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.578389 2549 raft_consensus.cc:2743] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.578418 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:26.578609 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:26.579192 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:26.579550 2289 raft_consensus.cc:2798] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:26.579773 2289 raft_consensus.cc:695] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:26.580044 2289 consensus_queue.cc:237] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.580255 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:26.580889 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:26.584379 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:26.584390 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 135) took 2418 ms. Trace:
I20250110 02:51:26.584458 2352 rpcz_store.cc:276] 0110 02:51:24.165764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.165804 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:26.584263 (+2418459us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:26.584271 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.584349 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2418213}
I20250110 02:51:26.584885 1441 catalog_manager.cc:5526] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:26.586285 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:26.596959 1181 tablet_bootstrap.cc:654] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:26.595485 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:26.603942 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:26.605414 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 136) took 2439 ms. Trace:
I20250110 02:51:26.605835 2352 rpcz_store.cc:276] 0110 02:51:24.166168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166213 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:26.605296 (+2439083us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:26.605305 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.605397 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2438963}
W20250110 02:51:26.606684 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d4f03a8c66446ea7c6d178cc7ef80a already in progress: creating tablet
I20250110 02:51:26.607059 2352 tablet_replica.cc:331] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.607589 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f162de87044005bdafe93292ff71b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.608047 2352 raft_consensus.cc:2238] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.610590 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f268832c914b88958d6336aa8813af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.614622 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23dff2b3a99464798847aa35dc5f584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.618654 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a721324b6a4ffc92bc2f73adccf0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.618916 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d064a7e8ea34f18b42407903daf7197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.523489 2224 rpcz_store.cc:276] 0110 02:51:23.385896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.385944 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:26.480044 (+3094100us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:26.480056 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.520116 (+ 40060us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.522665 (+ 2549us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.523387 (+ 722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3079036}
I20250110 02:51:26.621399 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 121) took 3230 ms. Trace:
I20250110 02:51:26.621500 2224 rpcz_store.cc:276] 0110 02:51:23.391379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.392489 (+ 1110us) service_pool.cc:225] Handling call
0110 02:51:26.621267 (+3228778us) ts_tablet_manager.cc:1145] Deleting tablet dba0cb7970f1453dac256b255ed7cac0
0110 02:51:26.621275 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.621355 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219832}
I20250110 02:51:26.622825 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2d9c4bce3419a9d8f6f9808549e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.623332 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 122) took 3219 ms. Trace:
I20250110 02:51:26.623425 2224 rpcz_store.cc:276] 0110 02:51:23.403857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.403920 (+ 63us) service_pool.cc:225] Handling call
0110 02:51:26.623212 (+3219292us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:26.623220 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.623288 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219113}
I20250110 02:51:26.626621 2224 tablet_replica.cc:331] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.626708 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8825527c0cf4b889a4f7f5e5faa2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.626864 2224 raft_consensus.cc:2238] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.627018 2224 raft_consensus.cc:2267] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:26.608140 853 consensus_peers.cc:487] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.630095 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471862cae972410b9770f7a8742cac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:26.630839 2352 raft_consensus.cc:2267] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.631367 2352 ts_tablet_manager.cc:1905] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.634147 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7d4422eca5604d149e73c2a669d07af0 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
W20250110 02:51:26.634434 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
W20250110 02:51:26.634845 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00e97e496b54dcf82b7182591d6f4aa already in progress: creating tablet
I20250110 02:51:26.635680 1188 tablet_bootstrap.cc:700] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Time spent opening tablet: real 0.137s user 0.001s sys 0.001s
I20250110 02:51:26.635769 1184 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.635896 1184 ts_tablet_manager.cc:1397] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.000s
I20250110 02:51:26.635922 1188 tablet_bootstrap.cc:654] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.636471 2224 ts_tablet_manager.cc:1905] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.636631 1184 raft_consensus.cc:357] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.636809 1184 raft_consensus.cc:383] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.636866 1184 raft_consensus.cc:738] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.637143 1184 consensus_queue.cc:260] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.637909 1184 ts_tablet_manager.cc:1428] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:26.639927 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:26.641817 2524 consensus_queue.cc:1035] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.646715 1181 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.646874 1181 ts_tablet_manager.cc:1397] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.128s user 0.002s sys 0.002s
I20250110 02:51:26.649550 1188 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.649685 1188 ts_tablet_manager.cc:1397] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.153s user 0.001s sys 0.001s
I20250110 02:51:26.650313 1188 raft_consensus.cc:357] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.650509 1188 raft_consensus.cc:383] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.650560 1188 raft_consensus.cc:738] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.650744 1188 consensus_queue.cc:260] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.651161 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:26.651302 1188 ts_tablet_manager.cc:1428] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:26.647719 1181 raft_consensus.cc:357] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.652284 1181 raft_consensus.cc:383] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.652361 1181 raft_consensus.cc:738] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.653499 1181 consensus_queue.cc:260] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.654528 1181 ts_tablet_manager.cc:1428] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:26.663028 1181 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.666254 1181 tablet_bootstrap.cc:654] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.666651 1184 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.666944 2549 raft_consensus.cc:491] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.667085 2549 raft_consensus.cc:513] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.667769 2549 leader_election.cc:290] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.668206 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.669373 711 leader_election.cc:343] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.669642 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.669780 1188 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:26.670162 709 leader_election.cc:343] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.670235 1184 tablet_bootstrap.cc:654] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.670260 709 leader_election.cc:304] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.670481 2549 raft_consensus.cc:2743] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.686064 2549 consensus_queue.cc:1035] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.689699 2549 raft_consensus.cc:491] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.689837 2549 raft_consensus.cc:513] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.690459 2549 leader_election.cc:290] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.690819 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.691286 711 leader_election.cc:343] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.691440 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:26.691824 709 leader_election.cc:343] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.691911 709 leader_election.cc:304] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.692265 2549 raft_consensus.cc:2743] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.693585 2224 ts_tablet_manager.cc:1918] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.699651 2345 consensus_queue.cc:1035] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.702414 1184 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.702543 1184 ts_tablet_manager.cc:1397] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20250110 02:51:26.703111 1184 raft_consensus.cc:357] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.703814 1184 raft_consensus.cc:383] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.703933 1184 raft_consensus.cc:738] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.704154 1184 consensus_queue.cc:260] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.704792 1184 ts_tablet_manager.cc:1428] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:26.705132 1184 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.707242 2224 log.cc:1198] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:26.707558 1184 tablet_bootstrap.cc:654] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.707787 2224 ts_tablet_manager.cc:1939] T a7397e41f7374e9fb1621dbcf30af703 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.708585 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 124) took 3304 ms. Trace:
I20250110 02:51:26.710037 2224 rpcz_store.cc:276] 0110 02:51:23.404341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.424703 (+ 20362us) service_pool.cc:225] Handling call
0110 02:51:26.626561 (+3201858us) ts_tablet_manager.cc:1145] Deleting tablet a7397e41f7374e9fb1621dbcf30af703
0110 02:51:26.626569 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.661130 (+ 34561us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.693463 (+ 32333us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.708566 (+ 15103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180695}
I20250110 02:51:26.710230 2357 ts_tablet_manager.cc:1918] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.709486 1181 tablet_bootstrap.cc:492] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.710726 1181 ts_tablet_manager.cc:1397] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20250110 02:51:26.711367 1181 raft_consensus.cc:357] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.711529 1181 raft_consensus.cc:383] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.711581 1181 raft_consensus.cc:738] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.711763 1181 consensus_queue.cc:260] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.712188 1181 ts_tablet_manager.cc:1428] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:26.712409 1181 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.713589 1181 tablet_bootstrap.cc:654] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.713866 1188 tablet_bootstrap.cc:654] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.714412 2224 tablet_replica.cc:331] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.714560 2357 log.cc:1198] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/24c4042c38de4969889f4821176fc7bb
I20250110 02:51:26.714526 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a7397e41f7374e9fb1621dbcf30af703 (table default.Table_94cf6a20190647dd87d3616b069b0243 [id=a91e19e65fd74454b4b4d339a5cbe806]) successfully deleted
I20250110 02:51:26.714664 2224 raft_consensus.cc:2238] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.714977 2224 raft_consensus.cc:2267] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:26.714855 711 consensus_peers.cc:487] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e25f9643ac0476a90c93fb9aeb921a3. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.716135 2357 ts_tablet_manager.cc:1939] T 24c4042c38de4969889f4821176fc7bb P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.716609 2224 ts_tablet_manager.cc:1905] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.717564 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 100) took 2504 ms. Trace:
I20250110 02:51:26.732748 2357 rpcz_store.cc:276] 0110 02:51:24.212714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.213725 (+ 1011us) service_pool.cc:225] Handling call
0110 02:51:26.506319 (+2292594us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:26.506328 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.638043 (+131715us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.710134 (+ 72091us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.717548 (+ 7414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2291082}
I20250110 02:51:26.715749 2352 ts_tablet_manager.cc:1918] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.733131 2352 log.cc:1198] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:26.733541 2352 ts_tablet_manager.cc:1939] T 5e183122d13a4c38988bf2fe6f7a3694 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.733916 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 24c4042c38de4969889f4821176fc7bb (table default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]) successfully deleted
I20250110 02:51:26.734951 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 101) took 2521 ms. Trace:
I20250110 02:51:26.738436 2357 rpcz_store.cc:276] 0110 02:51:24.213450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.213803 (+ 353us) service_pool.cc:225] Handling call
0110 02:51:26.734843 (+2521040us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:26.734852 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.734934 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2519580}
I20250110 02:51:26.738838 2224 ts_tablet_manager.cc:1918] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.738938 2224 log.cc:1198] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:26.739418 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 137) took 2572 ms. Trace:
W20250110 02:51:26.739797 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d4f03a8c66446ea7c6d178cc7ef80a already in progress: creating tablet
I20250110 02:51:26.740216 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 97) took 2527 ms. Trace:
I20250110 02:51:26.740334 2352 rpcz_store.cc:276] 0110 02:51:24.166526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166570 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:26.607018 (+2440448us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:26.607026 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.660703 (+ 53677us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.715668 (+ 54965us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.739400 (+ 23732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2440321}
I20250110 02:51:26.740581 2357 rpcz_store.cc:276] 0110 02:51:24.212466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.214882 (+ 2416us) service_pool.cc:225] Handling call
0110 02:51:26.740100 (+2525218us) ts_tablet_manager.cc:1145] Deleting tablet 3fb805b3e537466ab7fe0185421ad5aa
0110 02:51:26.740109 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.740200 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524703}
W20250110 02:51:26.741035 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:26.741498 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 5e183122d13a4c38988bf2fe6f7a3694 (table default.Table_eb30b4d9c27e4b5788054f7205e5610a [id=b65eed4420ef447989332175e0aefbc9]) successfully deleted
I20250110 02:51:26.741782 2357 tablet_replica.cc:331] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.742027 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 138) took 2575 ms. Trace:
I20250110 02:51:26.742305 2357 raft_consensus.cc:2238] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.742424 2352 rpcz_store.cc:276] 0110 02:51:24.166807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.166850 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:26.741928 (+2575078us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:26.741935 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.742012 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574924}
I20250110 02:51:26.742695 2357 raft_consensus.cc:2267] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:26.743022 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
I20250110 02:51:26.743366 2352 tablet_replica.cc:331] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.743865 2357 ts_tablet_manager.cc:1905] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.744110 2352 raft_consensus.cc:2238] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.744532 2352 raft_consensus.cc:2267] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.745886 2224 ts_tablet_manager.cc:1939] T 9e72ed2d31a8467f8c092b48d593c1cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.746857 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 125) took 3300 ms. Trace:
I20250110 02:51:26.747244 2352 ts_tablet_manager.cc:1905] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.752822 2224 rpcz_store.cc:276] 0110 02:51:23.446883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.446933 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:26.714339 (+3267406us) ts_tablet_manager.cc:1145] Deleting tablet 9e72ed2d31a8467f8c092b48d593c1cc
0110 02:51:26.714349 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.720135 (+ 5786us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.738762 (+ 18627us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.746839 (+ 8077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254221}
I20250110 02:51:26.753221 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 9e72ed2d31a8467f8c092b48d593c1cc (table default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]) successfully deleted
I20250110 02:51:26.753794 1188 tablet_bootstrap.cc:492] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.753911 1188 ts_tablet_manager.cc:1397] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20250110 02:51:26.753823 2224 tablet_replica.cc:331] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.754257 2224 raft_consensus.cc:2238] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.754617 2224 raft_consensus.cc:2267] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.754539 1188 raft_consensus.cc:357] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.754707 1188 raft_consensus.cc:383] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.754768 1188 raft_consensus.cc:738] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.754971 1188 consensus_queue.cc:260] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.755522 1188 ts_tablet_manager.cc:1428] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.755976 2224 ts_tablet_manager.cc:1905] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.758746 1188 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.765197 2224 ts_tablet_manager.cc:1918] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.766916 1188 tablet_bootstrap.cc:654] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.767946 1184 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.768095 1184 ts_tablet_manager.cc:1397] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.001s
I20250110 02:51:26.768621 1181 tablet_bootstrap.cc:492] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.768733 1184 raft_consensus.cc:357] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.768936 1184 raft_consensus.cc:383] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.768999 1184 raft_consensus.cc:738] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.769160 1181 ts_tablet_manager.cc:1397] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.007s
I20250110 02:51:26.769215 1184 consensus_queue.cc:260] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.769688 1184 ts_tablet_manager.cc:1428] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:26.769937 1184 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.770073 1181 raft_consensus.cc:357] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.770289 1181 raft_consensus.cc:383] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.770397 1181 raft_consensus.cc:738] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.771073 1181 consensus_queue.cc:260] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.771540 1181 ts_tablet_manager.cc:1428] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.771914 1181 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.773227 1181 tablet_bootstrap.cc:654] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.773870 2224 log.cc:1198] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:26.774291 2224 ts_tablet_manager.cc:1939] T cf711522866d46439d7ee69aa63d7f39 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.778426 1184 tablet_bootstrap.cc:654] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.778713 2352 ts_tablet_manager.cc:1918] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.778833 2352 log.cc:1198] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:26.780047 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 128) took 3318 ms. Trace:
I20250110 02:51:26.780167 2224 rpcz_store.cc:276] 0110 02:51:23.461919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.461957 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:26.753779 (+3291822us) ts_tablet_manager.cc:1145] Deleting tablet cf711522866d46439d7ee69aa63d7f39
0110 02:51:26.753789 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.761500 (+ 7711us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.765128 (+ 3628us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.780032 (+ 14904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3282541}
I20250110 02:51:26.780643 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet cf711522866d46439d7ee69aa63d7f39 (table default.Table_08d86300a7254466a9bff50b3f333765 [id=b0585ee264c44482941397f09c2eebb9]) successfully deleted
I20250110 02:51:26.781010 2224 tablet_replica.cc:331] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.781212 2224 raft_consensus.cc:2238] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.781594 2224 raft_consensus.cc:2267] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.785552 2224 ts_tablet_manager.cc:1905] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.788878 2357 ts_tablet_manager.cc:1918] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.790467 2357 log.cc:1198] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:26.790927 2357 ts_tablet_manager.cc:1939] T d20177229f7b4f11875e0007afb1dd00 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.791702 1188 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.791859 1188 ts_tablet_manager.cc:1397] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250110 02:51:26.792639 1188 raft_consensus.cc:357] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.792857 1188 raft_consensus.cc:383] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.792948 1188 raft_consensus.cc:738] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.793213 1188 consensus_queue.cc:260] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.793803 1188 ts_tablet_manager.cc:1428] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.794061 1188 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.795115 1181 tablet_bootstrap.cc:492] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.795291 1181 ts_tablet_manager.cc:1397] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250110 02:51:26.795651 1188 tablet_bootstrap.cc:654] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.796048 1181 raft_consensus.cc:357] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.796254 1181 raft_consensus.cc:383] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.796334 1181 raft_consensus.cc:738] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.796556 1181 consensus_queue.cc:260] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.776522 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:26.797705 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:26.798086 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:26.798583 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:26.799237 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:26.800137 1181 ts_tablet_manager.cc:1428] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:26.800137 2352 ts_tablet_manager.cc:1939] T 2ef72e05f8ed4d46b759c7a04497653b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.800402 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 139) took 2633 ms. Trace:
I20250110 02:51:26.800441 1181 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.801004 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 109) took 2574 ms. Trace:
I20250110 02:51:26.801119 2357 rpcz_store.cc:276] 0110 02:51:24.226957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.226999 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:26.741742 (+2514743us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:26.741750 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.763987 (+ 22237us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.788783 (+ 24796us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.800986 (+ 12203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2514522}
I20250110 02:51:26.801578 1181 tablet_bootstrap.cc:654] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.804590 2352 rpcz_store.cc:276] 0110 02:51:24.167149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.167189 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:26.743315 (+2576126us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:26.743322 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.775894 (+ 32572us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.778618 (+ 2724us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.800389 (+ 21771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575954}
I20250110 02:51:26.804940 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 104) took 2578 ms. Trace:
I20250110 02:51:26.805452 2352 tablet_replica.cc:331] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.805019 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d20177229f7b4f11875e0007afb1dd00 (table default.Table_d62161a3f0a2401198ea6728f350a9f6 [id=8f9113a41b724ee39fef09a6cba5a173]) successfully deleted
I20250110 02:51:26.805282 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2ef72e05f8ed4d46b759c7a04497653b (table default.Table_e298b60e8778429bb52cf546d58fa2d9 [id=62df87bfc41d477fb6044eff0ed5c3ea]) successfully deleted
I20250110 02:51:26.806038 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:26.806277 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:26.806298 2357 rpcz_store.cc:276] 0110 02:51:24.226527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.227106 (+ 579us) service_pool.cc:225] Handling call
0110 02:51:26.804828 (+2577722us) ts_tablet_manager.cc:1145] Deleting tablet f9b379b357194d729108428cbaebd51a
0110 02:51:26.804840 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.804925 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2577297}
I20250110 02:51:26.806502 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:26.806684 2352 raft_consensus.cc:2238] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.806906 2357 tablet_replica.cc:331] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.807057 2352 raft_consensus.cc:2267] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:26.807075 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:26.807271 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:26.807289 2357 raft_consensus.cc:2238] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.807412 2357 raft_consensus.cc:2267] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.807632 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:26.807926 2352 ts_tablet_manager.cc:1905] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.808143 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:26.808872 2357 ts_tablet_manager.cc:1905] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.809744 2224 ts_tablet_manager.cc:1918] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.810083 2224 log.cc:1198] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/9507a5283d8f4d07a293ba474c642298
I20250110 02:51:26.810909 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:26.811153 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:26.811348 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:26.810645 1184 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.812119 1184 ts_tablet_manager.cc:1397] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20250110 02:51:26.812800 1184 raft_consensus.cc:357] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.812965 1184 raft_consensus.cc:383] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.813009 1184 raft_consensus.cc:738] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.813213 1184 consensus_queue.cc:260] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.813665 1184 ts_tablet_manager.cc:1428] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:26.813880 1184 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.814765 2224 ts_tablet_manager.cc:1939] T 9507a5283d8f4d07a293ba474c642298 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.815183 1184 tablet_bootstrap.cc:654] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.815173 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:26.815335 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:26.815626 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:26.815640 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 130) took 3341 ms. Trace:
I20250110 02:51:26.815707 2224 rpcz_store.cc:276] 0110 02:51:23.473872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.473914 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:26.780969 (+3307055us) ts_tablet_manager.cc:1145] Deleting tablet 9507a5283d8f4d07a293ba474c642298
0110 02:51:26.780979 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.793212 (+ 12233us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.809655 (+ 16443us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.815625 (+ 5970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306887}
I20250110 02:51:26.816593 1188 tablet_bootstrap.cc:492] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.816717 1188 ts_tablet_manager.cc:1397] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250110 02:51:26.816850 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 9507a5283d8f4d07a293ba474c642298 (table default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]) successfully deleted
I20250110 02:51:26.817371 1188 raft_consensus.cc:357] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.817541 1188 raft_consensus.cc:383] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.817598 1188 raft_consensus.cc:738] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.817514 1181 tablet_bootstrap.cc:492] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.817786 1181 ts_tablet_manager.cc:1397] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250110 02:51:26.817783 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:26.817795 1188 consensus_queue.cc:260] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.818598 1181 raft_consensus.cc:357] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.818835 1181 raft_consensus.cc:383] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.818933 1181 raft_consensus.cc:738] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.819211 1181 consensus_queue.cc:260] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.822225 1181 ts_tablet_manager.cc:1428] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:26.822424 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 131) took 3348 ms. Trace:
I20250110 02:51:26.822573 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:26.822543 2224 rpcz_store.cc:276] 0110 02:51:23.474218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.474257 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:26.816580 (+3342323us) ts_tablet_manager.cc:1145] Deleting tablet 08705d260c0a462b94ba6b0dc5121123
0110 02:51:26.816587 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.822410 (+ 5823us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340603}
W20250110 02:51:26.823119 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08705d260c0a462b94ba6b0dc5121123 already in progress: creating tablet
I20250110 02:51:26.823885 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:26.824772 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:26.825028 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:26.825237 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:26.829104 2289 consensus_queue.cc:1035] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.830155 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:26.831027 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:26.831416 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:26.832370 1188 ts_tablet_manager.cc:1428] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250110 02:51:26.832620 1188 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.833251 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:26.833681 1181 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.834091 1188 tablet_bootstrap.cc:654] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.835211 1181 tablet_bootstrap.cc:654] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.836001 2224 tablet_replica.cc:331] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.836198 2224 raft_consensus.cc:2238] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.836534 2224 raft_consensus.cc:2267] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.837097 2224 ts_tablet_manager.cc:1905] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.840550 2289 consensus_queue.cc:1035] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.842255 2352 ts_tablet_manager.cc:1918] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.843664 2224 ts_tablet_manager.cc:1918] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.844281 2224 log.cc:1198] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/585320c0970c4097b195caaae33789b5
I20250110 02:51:26.844729 2224 ts_tablet_manager.cc:1939] T 585320c0970c4097b195caaae33789b5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.845744 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 129) took 3372 ms. Trace:
I20250110 02:51:26.846530 2224 rpcz_store.cc:276] 0110 02:51:23.473771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.474625 (+ 854us) service_pool.cc:225] Handling call
0110 02:51:26.822845 (+3348220us) ts_tablet_manager.cc:1145] Deleting tablet 585320c0970c4097b195caaae33789b5
0110 02:51:26.835958 (+ 13113us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.840330 (+ 4372us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.843601 (+ 3271us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.845714 (+ 2113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3344726}
I20250110 02:51:26.847071 2224 tablet_replica.cc:331] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:26.847477 2224 raft_consensus.cc:2238] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.848016 2224 raft_consensus.cc:2267] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.848937 1184 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.849202 1184 ts_tablet_manager.cc:1397] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250110 02:51:26.849833 1184 raft_consensus.cc:357] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.851184 1184 raft_consensus.cc:383] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.851266 1184 raft_consensus.cc:738] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.848856 2224 ts_tablet_manager.cc:1905] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.851495 1184 consensus_queue.cc:260] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.851729 2600 raft_consensus.cc:491] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.851856 2600 raft_consensus.cc:513] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.852056 1184 ts_tablet_manager.cc:1428] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:26.852273 1184 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.852362 2600 leader_election.cc:290] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.854017 1184 tablet_bootstrap.cc:654] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.847219 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 585320c0970c4097b195caaae33789b5 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
I20250110 02:51:26.854636 2289 consensus_queue.cc:1035] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.854892 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
W20250110 02:51:26.855504 996 leader_election.cc:343] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb
I20250110 02:51:26.858877 2357 ts_tablet_manager.cc:1918] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.859063 2357 log.cc:1198] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:26.859441 2357 ts_tablet_manager.cc:1939] T e015a41a03f84c91a45e46e4b7b5e7a9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.859778 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 103) took 2633 ms. Trace:
I20250110 02:51:26.862550 2357 rpcz_store.cc:276] 0110 02:51:24.226421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.227011 (+ 590us) service_pool.cc:225] Handling call
0110 02:51:26.806871 (+2579860us) ts_tablet_manager.cc:1145] Deleting tablet e015a41a03f84c91a45e46e4b7b5e7a9
0110 02:51:26.806879 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.828608 (+ 21729us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.858764 (+ 30156us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.859763 (+ 999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2579546}
I20250110 02:51:26.864698 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e015a41a03f84c91a45e46e4b7b5e7a9 (table default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]) successfully deleted
I20250110 02:51:26.853543 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c4042c38de4969889f4821176fc7bb" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.866080 2352 log.cc:1198] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8c3ce95413df40369ee599176f91b892
I20250110 02:51:26.866436 994 leader_election.cc:304] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.866664 2352 ts_tablet_manager.cc:1939] T 8c3ce95413df40369ee599176f91b892 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.866842 2289 raft_consensus.cc:2743] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.866918 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 119) took 2705 ms. Trace:
I20250110 02:51:26.866977 2352 rpcz_store.cc:276] 0110 02:51:24.161536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.169256 (+ 7720us) service_pool.cc:225] Handling call
0110 02:51:26.805417 (+2636161us) ts_tablet_manager.cc:1145] Deleting tablet 8c3ce95413df40369ee599176f91b892
0110 02:51:26.805424 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.839626 (+ 34202us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.842186 (+ 2560us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.866907 (+ 24721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2636002}
I20250110 02:51:26.873250 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:26.873790 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_8f356d58321749be9aed0b015ff38d71 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 1 got 0
I20250110 02:51:26.874599 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:26.879415 2352 tablet_replica.cc:331] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.880538 2352 raft_consensus.cc:2238] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:26.880878 2352 raft_consensus.cc:2267] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.881124 2357 tablet_replica.cc:331] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.881633 2352 ts_tablet_manager.cc:1905] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.885125 2024 consensus_queue.cc:1035] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.885504 2357 raft_consensus.cc:2238] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.877575 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8c3ce95413df40369ee599176f91b892 (table default.Table_a95ba565dbf940f79ed6f8b2eff1e74b [id=0743ba60cabd49c7a2c5230109d764eb]) successfully deleted
I20250110 02:51:26.887667 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:26.887701 1181 tablet_bootstrap.cc:492] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.888743 2024 raft_consensus.cc:491] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.888876 2024 raft_consensus.cc:513] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.889536 2024 leader_election.cc:290] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:26.890436 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
W20250110 02:51:26.890767 996 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:26.891140 1181 ts_tablet_manager.cc:1397] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.008s
I20250110 02:51:26.881384 1184 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:26.893276 1184 ts_tablet_manager.cc:1397] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.003s
I20250110 02:51:26.893951 1184 raft_consensus.cc:357] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.894186 1184 raft_consensus.cc:383] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.894266 1184 raft_consensus.cc:738] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.894537 1184 consensus_queue.cc:260] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.895200 1184 ts_tablet_manager.cc:1428] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:26.895381 2357 raft_consensus.cc:2267] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.890000 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b382d634db9642df8e00c5ccaa5787a9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:26.893141 2345 raft_consensus.cc:491] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.895606 2345 raft_consensus.cc:513] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.896075 2357 ts_tablet_manager.cc:1905] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.896160 2345 leader_election.cc:290] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.896883 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9ead3175140028e584e1ccb4874de" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:26.896978 853 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:26.897080 1061 raft_consensus.cc:2388] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:26.893795 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
W20250110 02:51:26.897392 994 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:26.897485 994 leader_election.cc:304] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.897488 853 leader_election.cc:304] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.897684 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:26.897866 2024 raft_consensus.cc:2743] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.898427 2345 raft_consensus.cc:3054] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:26.899611 2224 ts_tablet_manager.cc:1918] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.899689 2224 log.cc:1198] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:26.900002 2224 ts_tablet_manager.cc:1939] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:26.900291 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 132) took 3424 ms. Trace:
I20250110 02:51:26.900344 2224 rpcz_store.cc:276] 0110 02:51:23.475660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.475714 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:26.847016 (+3371302us) ts_tablet_manager.cc:1145] Deleting tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
0110 02:51:26.847025 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.885635 (+ 38610us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.899526 (+ 13891us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.900280 (+ 754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3364125}
I20250110 02:51:26.900820 2345 raft_consensus.cc:2743] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.901100 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (table default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]) successfully deleted
I20250110 02:51:26.902227 1181 raft_consensus.cc:357] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.903081 1188 tablet_bootstrap.cc:492] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:26.903185 1188 ts_tablet_manager.cc:1397] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.001s
I20250110 02:51:26.903419 1181 raft_consensus.cc:383] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.903724 1188 raft_consensus.cc:357] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.903899 1188 raft_consensus.cc:383] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.903955 1188 raft_consensus.cc:738] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.904129 1188 consensus_queue.cc:260] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.903872 1181 raft_consensus.cc:738] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.904590 1188 ts_tablet_manager.cc:1428] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 02:51:26.904870 1184 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:26.906308 1181 consensus_queue.cc:260] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.907158 1184 tablet_bootstrap.cc:654] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:26.909605 853 consensus_peers.cc:487] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:26.910084 1181 ts_tablet_manager.cc:1428] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250110 02:51:26.910787 1188 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:26.910877 2345 raft_consensus.cc:491] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.911026 2345 raft_consensus.cc:513] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.911581 2345 leader_election.cc:290] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:26.912521 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:26.914260 1188 tablet_bootstrap.cc:654] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:26.914601 1444 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bd2283e7ed7456dae466a8582587eb7 consensus state has no leader
I20250110 02:51:26.914793 1444 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 87) took 3884 ms. Trace:
I20250110 02:51:26.914846 1181 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.914875 1444 rpcz_store.cc:276] 0110 02:51:23.031723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.987842 (+1956119us) service_pool.cc:225] Handling call
0110 02:51:25.032072 (+ 44230us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:25.054466 (+ 22394us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:25.054604 (+ 138us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:25.054662 (+ 58us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:25.077719 (+ 23057us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:26.914741 (+1837022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.057029 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.057425 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.057430 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.057432 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.057494 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.057501 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.057504 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.057541 (+ 37us) tablet.cc:698] Row locks acquired
0110 02:51:25.057543 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:25.057566 (+ 23us) write_op.cc:270] Start()
0110 02:51:25.057586 (+ 20us) write_op.cc:276] Timestamp: P: 1736477485057509 usec, L: 0
0110 02:51:25.057589 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.057696 (+ 107us) log.cc:844] Serialized 405 byte log entry
0110 02:51:25.077048 (+ 19352us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.077329 (+ 281us) write_op.cc:301] APPLY: starting
0110 02:51:25.077346 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.077385 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.077411 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.077463 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.077489 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.077495 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.077558 (+ 63us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.077628 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.077645 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:25.077653 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.077721 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.077724 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":235,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":388,"num_ops":1,"prepare.queue_time_us":1927,"prepare.run_cpu_time_us":438,"prepare.run_wall_time_us":2912,"raft.queue_time_us":2165,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":19430}]]}
W20250110 02:51:26.914804 853 leader_election.cc:343] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:26.919175 853 leader_election.cc:304] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.919921 2345 raft_consensus.cc:2743] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:26.926627 1449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_8c793fc0a0294bcf925860d30901b7d0 [id=bca53af91a9746b990b7329b337d993b] alter complete (version 1)
I20250110 02:51:26.926743 1181 tablet_bootstrap.cc:654] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.936344 2357 ts_tablet_manager.cc:1918] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:26.944547 2352 ts_tablet_manager.cc:1918] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:26.946162 1181 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.946345 1181 ts_tablet_manager.cc:1397] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20250110 02:51:26.947152 1181 raft_consensus.cc:357] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.948449 1449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_bfc621ed30834a669b0ba54dcda3b490 [id=1c37b52d30d44f5d9fbb15cca02d70b9] alter complete (version 3)
I20250110 02:51:26.949002 2352 log.cc:1198] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:26.949556 2352 ts_tablet_manager.cc:1939] T 60e534bf66f044b88d8cda0f3a1228f4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:26.949880 1181 raft_consensus.cc:383] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:26.949990 1181 raft_consensus.cc:738] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:26.950198 1181 consensus_queue.cc:260] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:26.950696 1181 ts_tablet_manager.cc:1428] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:26.950975 1181 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:26.956972 1181 tablet_bootstrap.cc:654] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:26.957058 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 140) took 2764 ms. Trace:
I20250110 02:51:26.957887 2352 rpcz_store.cc:276] 0110 02:51:24.192394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.192449 (+ 55us) service_pool.cc:225] Handling call
0110 02:51:26.879371 (+2686922us) ts_tablet_manager.cc:1145] Deleting tablet 60e534bf66f044b88d8cda0f3a1228f4
0110 02:51:26.879380 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.907128 (+ 27748us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.944457 (+ 37329us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.957040 (+ 12583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2686599}
I20250110 02:51:26.958850 2357 log.cc:1198] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:26.960553 2352 tablet_replica.cc:331] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:26.960772 2352 raft_consensus.cc:2238] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.961179 2357 ts_tablet_manager.cc:1939] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:26.961472 2352 raft_consensus.cc:2267] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.962716 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 105) took 2736 ms. Trace:
I20250110 02:51:26.963284 2352 ts_tablet_manager.cc:1905] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.963824 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 60e534bf66f044b88d8cda0f3a1228f4 (table default.Table_8df25635dd7e4d58818dc14fc678e221 [id=4893b36e085445c4a454cf47ebe1955b]) successfully deleted
I20250110 02:51:26.964565 1456 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7] alter complete (version 4)
W20250110 02:51:26.966328 1450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09c029deb4c54ce68d560059020f0bba consensus state has no leader
I20250110 02:51:26.966470 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_3f8f54a3c0194e228523d98131042d67 [id=dfb29df1e3c54611abc07a3326df3215] alter complete (version 1)
I20250110 02:51:26.975915 2357 rpcz_store.cc:276] 0110 02:51:24.226610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.227262 (+ 652us) service_pool.cc:225] Handling call
0110 02:51:26.881085 (+2653823us) ts_tablet_manager.cc:1145] Deleting tablet 6e0b2ec1d5814144bc58261e6c6e8d35
0110 02:51:26.881093 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:26.933017 (+ 51924us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.936253 (+ 3236us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:26.962698 (+ 26445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653715}
I20250110 02:51:26.977125 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (table default.Table_c2f177ce9d6a420bbc77b33929ccaa7a [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]) successfully deleted
I20250110 02:51:26.977268 2357 tablet_replica.cc:331] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:26.977458 2357 raft_consensus.cc:2238] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:26.978104 2357 raft_consensus.cc:2267] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:26.980551 2357 ts_tablet_manager.cc:1905] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:26.985306 2377 raft_consensus.cc:491] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:26.985442 2377 raft_consensus.cc:513] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.986025 2377 leader_election.cc:290] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:26.987270 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
W20250110 02:51:26.987457 853 leader_election.cc:343] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:26.987541 853 leader_election.cc:304] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:26.987859 2632 raft_consensus.cc:491] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85f9597077934f1a9814d284ed06d13d)
I20250110 02:51:26.987991 2632 raft_consensus.cc:513] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.988552 2632 leader_election.cc:290] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:26.989487 2632 raft_consensus.cc:491] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85f9597077934f1a9814d284ed06d13d)
I20250110 02:51:26.989598 2632 raft_consensus.cc:513] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:26.990121 2632 leader_election.cc:290] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:26.991238 709 leader_election.cc:343] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:26.991550 2378 raft_consensus.cc:2743] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:26.992290 711 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: b43bb790e85343308a1bb1e1110bfb9c
W20250110 02:51:26.992642 711 leader_election.cc:343] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:26.992722 711 leader_election.cc:304] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.993382 2632 raft_consensus.cc:2743] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:26.993405 1056 raft_consensus.cc:3054] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:26.994527 709 leader_election.cc:343] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:26.994630 709 leader_election.cc:304] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:26.995050 2632 raft_consensus.cc:2743] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:26.996243 2377 consensus_queue.cc:1035] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:26.998195 1181 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:26.998960 1181 ts_tablet_manager.cc:1397] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.015s
I20250110 02:51:26.999809 1181 raft_consensus.cc:357] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.000936 1181 raft_consensus.cc:383] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.001099 1181 raft_consensus.cc:738] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.001433 1181 consensus_queue.cc:260] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.002254 1181 ts_tablet_manager.cc:1428] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:27.005191 1441 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_0c2624c5c6b646e1969b7e0abedf82e0 [id=6f37f6d6318e45f1a016b8a34634050d] alter complete (version 1)
I20250110 02:51:27.013706 1449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_f31fd3b9463a49f9b3dd78cc630dfdb7 [id=00766890e5f048ceb05d2732814df989] alter complete (version 1)
I20250110 02:51:27.018026 2352 ts_tablet_manager.cc:1918] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.028851 2352 log.cc:1198] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/971f97a3be9b49958b1218a275e126d4
I20250110 02:51:27.029986 2357 ts_tablet_manager.cc:1918] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.030429 2352 ts_tablet_manager.cc:1939] T 971f97a3be9b49958b1218a275e126d4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.033159 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 142) took 2835 ms. Trace:
I20250110 02:51:27.033610 2352 rpcz_store.cc:276] 0110 02:51:24.197771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.197809 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:26.958652 (+2760843us) ts_tablet_manager.cc:1145] Deleting tablet 971f97a3be9b49958b1218a275e126d4
0110 02:51:26.960516 (+ 1864us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.001556 (+ 41040us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.017906 (+ 16350us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.033144 (+ 15238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2760689}
I20250110 02:51:27.034237 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 971f97a3be9b49958b1218a275e126d4 (table default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]) successfully deleted
I20250110 02:51:27.034339 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 141) took 2836 ms. Trace:
I20250110 02:51:27.034413 2352 rpcz_store.cc:276] 0110 02:51:24.197641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.197722 (+ 81us) service_pool.cc:225] Handling call
0110 02:51:27.034237 (+2836515us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:27.034245 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.034326 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2835590}
I20250110 02:51:27.034687 2357 log.cc:1198] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:27.035112 2357 ts_tablet_manager.cc:1939] T 76833ce515cf4ee9bc8fff0f626af220 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.037650 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 146) took 2774 ms. Trace:
W20250110 02:51:27.037729 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:27.038749 2352 rpcz_store.cc:276] 0110 02:51:24.263100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.263150 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:27.037559 (+2774409us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:27.037567 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.037637 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774221}
I20250110 02:51:27.039469 2352 tablet_replica.cc:331] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:27.039462 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:27.039870 2352 raft_consensus.cc:2238] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.040690 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 108) took 2813 ms. Trace:
I20250110 02:51:27.040948 2352 raft_consensus.cc:2267] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.041106 2357 rpcz_store.cc:276] 0110 02:51:24.226878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.226980 (+ 102us) service_pool.cc:225] Handling call
0110 02:51:26.977228 (+2750248us) ts_tablet_manager.cc:1145] Deleting tablet 76833ce515cf4ee9bc8fff0f626af220
0110 02:51:26.977237 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.006533 (+ 29296us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.029892 (+ 23359us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.040658 (+ 10766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749659}
I20250110 02:51:27.041795 2352 ts_tablet_manager.cc:1905] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.042181 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 76833ce515cf4ee9bc8fff0f626af220 (table default.Table_9cdcc4b295e54e45a928a008a61f0285 [id=32a63f537c9f47e493da771eebd5b8e6]) successfully deleted
I20250110 02:51:27.042690 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 107) took 2815 ms. Trace:
I20250110 02:51:27.042793 1441 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_6e6ef42caec74748b566727e29ff4336 [id=7d20fea78e494fe3bc3631bb749d58eb] alter complete (version 1)
I20250110 02:51:27.044975 2357 rpcz_store.cc:276] 0110 02:51:24.226803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.227418 (+ 615us) service_pool.cc:225] Handling call
0110 02:51:27.042588 (+2815170us) ts_tablet_manager.cc:1145] Deleting tablet b4f1782c46a44ed091f4c533226c71f2
0110 02:51:27.042596 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.042675 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2814294}
I20250110 02:51:27.044775 2524 raft_consensus.cc:491] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.045482 2524 raft_consensus.cc:513] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.046232 2524 leader_election.cc:290] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.046782 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.046977 1056 raft_consensus.cc:2463] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.047439 853 leader_election.cc:304] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.047636 2524 raft_consensus.cc:2798] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.047695 2524 raft_consensus.cc:491] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.047755 2524 raft_consensus.cc:3054] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.050173 2524 raft_consensus.cc:513] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.051003 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.051143 1056 raft_consensus.cc:3054] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.051882 2524 leader_election.cc:290] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.053431 1056 raft_consensus.cc:2463] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.053759 853 leader_election.cc:304] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.054006 2524 raft_consensus.cc:2798] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.054332 2524 raft_consensus.cc:695] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.054641 2524 consensus_queue.cc:237] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.058035 1457 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_c9035b932c4f40428820b9bfa2d4ca61 [id=a68859689efc4362ab519c103dfc42a0] alter complete (version 2)
I20250110 02:51:27.065237 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_4b25e890ab5a4e4889f28402f205b95f [id=58c2b267860e44998d49c1905758f326] alter complete (version 2)
I20250110 02:51:27.047138 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.065964 764 raft_consensus.cc:2463] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.051728 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c7a313ba94e5ea85e45276658ba04" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.066277 772 raft_consensus.cc:3054] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.057441 2632 raft_consensus.cc:491] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.066607 2632 raft_consensus.cc:513] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.067294 2632 leader_election.cc:290] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.067677 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.068150 711 leader_election.cc:343] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.068480 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.068954 709 leader_election.cc:343] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.068917 1457 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_72a27bbc01334e52ae01e2c46eab0581 [id=015845fa33aa4443b36402d79b4c94bf] alter complete (version 2)
I20250110 02:51:27.069044 709 leader_election.cc:304] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.069380 2632 raft_consensus.cc:2743] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.070780 772 raft_consensus.cc:2463] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.071933 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 106) took 2845 ms. Trace:
W20250110 02:51:27.072023 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:27.073333 2357 rpcz_store.cc:276] 0110 02:51:24.226708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.229106 (+ 2398us) service_pool.cc:225] Handling call
0110 02:51:27.071842 (+2842736us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:27.071851 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.071915 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2842574}
W20250110 02:51:27.073776 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because the tablet was not found. No further retry: Not found: Tablet not found: 0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:27.073868 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 102) took 2847 ms. Trace:
I20250110 02:51:27.073935 2357 rpcz_store.cc:276] 0110 02:51:24.226300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.229289 (+ 2989us) service_pool.cc:225] Handling call
0110 02:51:27.073742 (+2844453us) ts_tablet_manager.cc:1145] Deleting tablet 940445840db3441598eba57d3ccb8c9c
0110 02:51:27.073750 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.073853 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844330}
W20250110 02:51:27.074261 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because the tablet was not found. No further retry: Not found: Tablet not found: 940445840db3441598eba57d3ccb8c9c
I20250110 02:51:27.074306 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 112) took 2825 ms. Trace:
I20250110 02:51:27.074414 2357 rpcz_store.cc:276] 0110 02:51:24.249102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.249154 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:27.074209 (+2825055us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:27.074217 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.074294 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824874}
W20250110 02:51:27.075451 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:27.075984 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 115) took 2826 ms. Trace:
I20250110 02:51:27.086436 2357 rpcz_store.cc:276] 0110 02:51:24.249424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.249460 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:27.075887 (+2826427us) ts_tablet_manager.cc:1145] Deleting tablet 82bd4307e5fe4f61b9308005cf3446e1
0110 02:51:27.075895 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.075968 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2826263}
W20250110 02:51:27.086971 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
I20250110 02:51:27.087306 2357 tablet_replica.cc:331] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.087538 2357 raft_consensus.cc:2238] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.087539 2524 raft_consensus.cc:491] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.087646 2524 raft_consensus.cc:513] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.087754 2357 raft_consensus.cc:2267] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.088393 2357 ts_tablet_manager.cc:1905] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.088640 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.088835 2524 leader_election.cc:290] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.088855 1056 raft_consensus.cc:2463] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.089244 853 leader_election.cc:304] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
W20250110 02:51:27.089527 853 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:27.090795 2524 raft_consensus.cc:2798] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.090874 2524 raft_consensus.cc:491] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.090938 2524 raft_consensus.cc:3054] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.093281 2524 raft_consensus.cc:513] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.094178 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9289de0630804f9aa60fdaf5f3f15f68" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.094357 1056 raft_consensus.cc:3054] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.096597 1056 raft_consensus.cc:2463] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.097048 853 leader_election.cc:304] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.097265 2378 raft_consensus.cc:2798] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.097384 2378 raft_consensus.cc:695] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.097612 2378 consensus_queue.cc:237] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.101364 1441 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_d6c1d16cc59d472dabdbb7be7378be29 [id=1d00dbb015e142329070adad93cf1bcb] alter complete (version 5)
I20250110 02:51:27.129235 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:27.129926 2524 leader_election.cc:290] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:27.130427 853 leader_election.cc:343] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:27.133145 1440 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_459a9c14d71c40e5a90a716b251f8be4 [id=7ed2fe5bf29d4f71941ee061577ebf26] alter complete (version 1)
I20250110 02:51:27.134518 1450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 102) took 2674 ms. Trace:
I20250110 02:51:27.134618 1450 rpcz_store.cc:276] 0110 02:51:24.460514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.375683 (+915169us) service_pool.cc:225] Handling call
0110 02:51:25.376026 (+ 343us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:25.376247 (+ 221us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:25.376408 (+ 161us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:25.376489 (+ 81us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:25.384789 (+ 8300us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:27.134459 (+1749670us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.377071 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.377159 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.377166 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.377168 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.377257 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.377267 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.377271 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.377337 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:25.377338 (+ 1us) write_op.cc:260] PREPARE: finished
0110 02:51:25.377366 (+ 28us) write_op.cc:270] Start()
0110 02:51:25.377390 (+ 24us) write_op.cc:276] Timestamp: P: 1736477485377330 usec, L: 0
0110 02:51:25.377393 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.377523 (+ 130us) log.cc:844] Serialized 405 byte log entry
0110 02:51:25.384119 (+ 6596us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.384222 (+ 103us) write_op.cc:301] APPLY: starting
0110 02:51:25.384236 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.384272 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.384302 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.384343 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.384371 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.384378 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:25.384438 (+ 60us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.384486 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.384500 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:25.384505 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.384563 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.384567 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":335,"apply.run_wall_time_us":335,"num_ops":1,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":540,"prepare.run_wall_time_us":540,"raft.queue_time_us":575,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":261,"replication_time_us":6701}]]}
I20250110 02:51:27.135349 1450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_6a8868f4b22043c193efdd181759b4f4" } new_table_name: "default.Table_ee66a0c06761457ba22cb964ce907675" modify_external_catalogs: true
I20250110 02:51:27.152362 2352 ts_tablet_manager.cc:1918] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.152483 2352 log.cc:1198] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:27.153604 1188 tablet_bootstrap.cc:492] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:27.153750 1188 ts_tablet_manager.cc:1397] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.243s user 0.014s sys 0.006s
I20250110 02:51:27.152179 1441 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_2b2430678ec9494e8179181b89afcd33 [id=b8fb31b88bef421ca915a74593e95dde] alter complete (version 1)
I20250110 02:51:27.154629 1188 raft_consensus.cc:357] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.154876 1188 raft_consensus.cc:383] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.154971 1188 raft_consensus.cc:738] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.155223 1188 consensus_queue.cc:260] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.155593 1184 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.155728 1188 ts_tablet_manager.cc:1428] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:27.155828 1184 ts_tablet_manager.cc:1397] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.251s user 0.001s sys 0.003s
I20250110 02:51:27.156050 1188 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:27.157052 1184 raft_consensus.cc:357] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.157797 1184 raft_consensus.cc:383] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.157954 1184 raft_consensus.cc:738] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.158412 1184 consensus_queue.cc:260] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.158861 1188 tablet_bootstrap.cc:654] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.159387 1184 ts_tablet_manager.cc:1428] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:27.162420 2352 ts_tablet_manager.cc:1939] T 8eb9ead3175140028e584e1ccb4874de P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.162846 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 149) took 2899 ms. Trace:
W20250110 02:51:27.154551 2504 log.cc:926] Time spent T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Append to log took a long time: real 0.240s user 0.001s sys 0.003s
I20250110 02:51:27.162974 2352 rpcz_store.cc:276] 0110 02:51:24.263430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.263473 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:27.039429 (+2775956us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:27.039437 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.088739 (+ 49302us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.152247 (+ 63508us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.162820 (+ 10573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775816}
I20250110 02:51:27.163481 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8eb9ead3175140028e584e1ccb4874de (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:27.163558 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 145) took 2900 ms. Trace:
I20250110 02:51:27.163615 2352 rpcz_store.cc:276] 0110 02:51:24.263001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.263714 (+ 713us) service_pool.cc:225] Handling call
0110 02:51:27.163452 (+2899738us) ts_tablet_manager.cc:1145] Deleting tablet 82bd4307e5fe4f61b9308005cf3446e1
0110 02:51:27.163462 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.163544 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899593}
I20250110 02:51:27.164227 2352 tablet_replica.cc:331] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:27.164213 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
I20250110 02:51:27.164467 2352 raft_consensus.cc:2238] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.164641 2352 raft_consensus.cc:2267] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.165059 1181 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.166728 1181 tablet_bootstrap.cc:654] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.171406 2224 tablet_replica.cc:331] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.171594 2600 consensus_queue.cc:1035] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.171763 2224 raft_consensus.cc:2238] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.171957 2224 raft_consensus.cc:2267] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.176163 2224 ts_tablet_manager.cc:1905] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.179100 2357 ts_tablet_manager.cc:1918] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.179219 2357 log.cc:1198] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:27.179512 2357 ts_tablet_manager.cc:1939] T c00e97e496b54dcf82b7182591d6f4aa P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.180228 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 117) took 2930 ms. Trace:
I20250110 02:51:27.180313 2357 rpcz_store.cc:276] 0110 02:51:24.249663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.249722 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:27.087258 (+2837536us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:27.087267 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.129571 (+ 42304us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.178993 (+ 49422us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.180205 (+ 1212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837411}
I20250110 02:51:27.180614 2357 tablet_replica.cc:331] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.180781 2357 raft_consensus.cc:2238] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.180805 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c00e97e496b54dcf82b7182591d6f4aa (table default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]) successfully deleted
I20250110 02:51:27.180944 2357 raft_consensus.cc:2267] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.181419 2357 ts_tablet_manager.cc:1905] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.182618 1184 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.191123 2352 ts_tablet_manager.cc:1905] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.192584 1181 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.192723 1181 ts_tablet_manager.cc:1397] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20250110 02:51:27.193588 1181 raft_consensus.cc:357] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.193847 1181 raft_consensus.cc:383] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.193939 1181 raft_consensus.cc:738] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.194186 1181 consensus_queue.cc:260] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.194787 1181 ts_tablet_manager.cc:1428] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:27.195139 1181 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.196753 1181 tablet_bootstrap.cc:654] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.203944 2024 raft_consensus.cc:491] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.204059 2024 raft_consensus.cc:513] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.204569 2024 leader_election.cc:290] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.205533 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.205701 772 raft_consensus.cc:2463] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.206127 996 leader_election.cc:304] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.206614 2024 raft_consensus.cc:2798] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.206699 2024 raft_consensus.cc:491] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.206632 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.206748 2024 raft_consensus.cc:3054] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.206817 913 raft_consensus.cc:2463] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.208460 1184 tablet_bootstrap.cc:654] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.210529 2024 raft_consensus.cc:513] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.211288 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:27.211431 913 raft_consensus.cc:3054] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.211862 2632 raft_consensus.cc:491] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.212013 2632 raft_consensus.cc:513] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.212749 2632 leader_election.cc:290] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.213317 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.213438 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ebf8debb9d4394839d1c1cb97820a5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.213527 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.213590 772 raft_consensus.cc:3054] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:27.213779 709 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.214074 913 raft_consensus.cc:2463] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.214200 2024 leader_election.cc:290] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:27.214308 711 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.214574 711 leader_election.cc:304] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.214970 2632 raft_consensus.cc:2743] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.217082 994 leader_election.cc:304] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.217638 2600 raft_consensus.cc:2798] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.217772 2600 raft_consensus.cc:695] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.218027 2600 consensus_queue.cc:237] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.222235 772 raft_consensus.cc:2463] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.225337 2632 raft_consensus.cc:491] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.225435 2632 raft_consensus.cc:513] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.225953 2632 leader_election.cc:290] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.226316 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.226733 711 leader_election.cc:343] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.227288 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.228140 709 leader_election.cc:343] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.228230 709 leader_election.cc:304] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.228169 2024 consensus_queue.cc:1035] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.228634 2632 raft_consensus.cc:2743] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.230259 1188 tablet_bootstrap.cc:492] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:27.232524 2024 raft_consensus.cc:491] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.232673 2024 raft_consensus.cc:513] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.233819 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.233996 764 raft_consensus.cc:2463] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.234359 996 leader_election.cc:304] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.234597 2456 raft_consensus.cc:2798] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.234652 2456 raft_consensus.cc:491] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.234699 2456 raft_consensus.cc:3054] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.234724 2024 leader_election.cc:290] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:27.236802 2456 raft_consensus.cc:513] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.237432 2456 leader_election.cc:290] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:27.237934 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.238132 764 raft_consensus.cc:3054] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.240708 764 raft_consensus.cc:2463] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.241117 996 leader_election.cc:304] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.241380 2456 raft_consensus.cc:2798] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.241518 2456 raft_consensus.cc:695] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.241731 2456 consensus_queue.cc:237] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.244235 1184 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
W20250110 02:51:27.245322 994 consensus_peers.cc:487] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 971f97a3be9b49958b1218a275e126d4. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.249558 1184 ts_tablet_manager.cc:1397] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.001s
I20250110 02:51:27.238646 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:27.250674 994 leader_election.cc:343] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.244889 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.251283 994 leader_election.cc:343] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.251609 1188 ts_tablet_manager.cc:1397] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.001s
I20250110 02:51:27.252699 1188 raft_consensus.cc:357] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.252954 1188 raft_consensus.cc:383] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.253057 1188 raft_consensus.cc:738] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.253556 1188 consensus_queue.cc:260] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.254137 1188 ts_tablet_manager.cc:1428] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:27.255471 2224 ts_tablet_manager.cc:1918] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.255941 1184 raft_consensus.cc:357] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.257004 1184 raft_consensus.cc:383] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.257198 1184 raft_consensus.cc:738] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.257604 1184 consensus_queue.cc:260] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.258709 2224 log.cc:1198] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:27.259171 1184 ts_tablet_manager.cc:1428] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250110 02:51:27.259630 2224 ts_tablet_manager.cc:1939] T 6e4d3270e3a04cc99f90943b9573ec7e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.261133 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 133) took 3775 ms. Trace:
I20250110 02:51:27.261422 1184 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.261708 2524 raft_consensus.cc:491] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.261895 2524 raft_consensus.cc:513] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.262840 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452223af5f6a4836a0f582d394a9453e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.263584 853 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
I20250110 02:51:27.263636 1056 raft_consensus.cc:2463] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.263700 2524 leader_election.cc:290] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.264040 1184 tablet_bootstrap.cc:654] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.264070 853 leader_election.cc:304] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.263562 2224 rpcz_store.cc:276] 0110 02:51:23.485413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.485561 (+ 148us) service_pool.cc:225] Handling call
0110 02:51:27.081339 (+3595778us) ts_tablet_manager.cc:1145] Deleting tablet 6e4d3270e3a04cc99f90943b9573ec7e
0110 02:51:27.171337 (+ 89998us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.218156 (+ 46819us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.255362 (+ 37206us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.261118 (+ 5756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595616}
I20250110 02:51:27.263315 1181 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.264504 1181 ts_tablet_manager.cc:1397] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.013s
I20250110 02:51:27.264544 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 6e4d3270e3a04cc99f90943b9573ec7e (table default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]) successfully deleted
I20250110 02:51:27.265321 2224 tablet_replica.cc:331] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.265444 2524 raft_consensus.cc:2798] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.265523 2224 raft_consensus.cc:2238] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.265545 2524 raft_consensus.cc:491] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.265661 2524 raft_consensus.cc:3054] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.265797 2224 raft_consensus.cc:2267] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.265239 1181 raft_consensus.cc:357] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.266644 1181 raft_consensus.cc:383] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.266705 1181 raft_consensus.cc:738] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.266892 1181 consensus_queue.cc:260] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.267333 1181 ts_tablet_manager.cc:1428] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:27.267957 1181 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.269397 2378 raft_consensus.cc:491] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.269519 1181 tablet_bootstrap.cc:654] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.269511 2378 raft_consensus.cc:513] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.270081 2378 leader_election.cc:290] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.270565 2224 ts_tablet_manager.cc:1905] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.270659 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.269910 2524 raft_consensus.cc:513] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.272732 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452223af5f6a4836a0f582d394a9453e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.272886 1056 raft_consensus.cc:3054] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.274363 2024 raft_consensus.cc:491] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.274541 2024 raft_consensus.cc:513] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.275095 2024 leader_election.cc:290] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.275446 1056 raft_consensus.cc:2463] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.275434 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.275583 764 raft_consensus.cc:2463] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.275862 853 leader_election.cc:304] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.275898 996 leader_election.cc:304] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.276108 2024 raft_consensus.cc:2798] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.276130 2378 raft_consensus.cc:2798] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.276230 2378 raft_consensus.cc:695] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.276234 2024 raft_consensus.cc:491] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.276289 2024 raft_consensus.cc:3054] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.276427 2378 consensus_queue.cc:237] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.278889 2024 raft_consensus.cc:513] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.280187 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.280325 764 raft_consensus.cc:3054] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.271867 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815767a4c7624852bfedab9fa6cb5f44" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.281438 772 raft_consensus.cc:2388] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:27.281747 853 leader_election.cc:304] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.282070 2345 raft_consensus.cc:3054] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.283175 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:27.283607 764 raft_consensus.cc:2463] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.283975 996 leader_election.cc:304] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.284045 2345 raft_consensus.cc:2743] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.284183 2024 leader_election.cc:290] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.284312 2024 raft_consensus.cc:2798] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.284837 2024 raft_consensus.cc:695] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.285063 2024 consensus_queue.cc:237] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.287886 853 consensus_peers.cc:487] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.289935 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:27.290400 994 leader_election.cc:343] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.290798 1188 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:27.298187 1184 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.299182 1184 ts_tablet_manager.cc:1397] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.002s
I20250110 02:51:27.301002 2224 ts_tablet_manager.cc:1918] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.301556 1184 raft_consensus.cc:357] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.302547 1184 raft_consensus.cc:383] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.302748 1184 raft_consensus.cc:738] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.303068 1184 consensus_queue.cc:260] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.304026 2524 leader_election.cc:290] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:27.304692 853 leader_election.cc:343] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
I20250110 02:51:27.305395 1184 ts_tablet_manager.cc:1428] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:27.305583 2224 log.cc:1198] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:27.297783 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ade70d7fe34776bef3c5853f8dde74" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.306499 994 leader_election.cc:343] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.306604 2224 ts_tablet_manager.cc:1939] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.307283 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 126) took 3856 ms. Trace:
I20250110 02:51:27.310339 2224 rpcz_store.cc:276] 0110 02:51:23.450960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.494459 (+ 43499us) service_pool.cc:225] Handling call
0110 02:51:27.265279 (+3770820us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:27.265287 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.280029 (+ 14742us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.300911 (+ 20882us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.307267 (+ 6356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":3770642}
I20250110 02:51:27.311097 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
I20250110 02:51:27.311475 2224 tablet_replica.cc:331] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.311873 2224 raft_consensus.cc:2238] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.312793 2224 raft_consensus.cc:2267] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.312743 2345 consensus_queue.cc:1035] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.314486 1181 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.314610 1181 ts_tablet_manager.cc:1397] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.002s
I20250110 02:51:27.315279 1181 raft_consensus.cc:357] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.315466 1181 raft_consensus.cc:383] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.315536 1181 raft_consensus.cc:738] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.315768 1181 consensus_queue.cc:260] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.315891 2024 raft_consensus.cc:491] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.316006 2024 raft_consensus.cc:513] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.316378 1181 ts_tablet_manager.cc:1428] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:27.316653 1181 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:27.317612 996 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:27.317842 2024 leader_election.cc:290] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.318115 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e05a7554a14805bdcbd1824a5bd9d5" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.318562 994 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.318625 2224 ts_tablet_manager.cc:1905] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.318652 994 leader_election.cc:304] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.320003 2456 raft_consensus.cc:2743] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.325596 1181 tablet_bootstrap.cc:654] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.329772 1184 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.334306 1184 tablet_bootstrap.cc:654] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.337306 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:27.339380 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:27.340703 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:27.341588 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:27.328562 2352 ts_tablet_manager.cc:1918] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.343211 2352 log.cc:1198] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/2e25f9643ac0476a90c93fb9aeb921a3
I20250110 02:51:27.343015 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:27.343580 2352 ts_tablet_manager.cc:1939] T 2e25f9643ac0476a90c93fb9aeb921a3 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.344364 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 147) took 3081 ms. Trace:
I20250110 02:51:27.344841 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
W20250110 02:51:27.339989 853 consensus_peers.cc:487] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.335335 1188 tablet_bootstrap.cc:654] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.347072 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:27.349352 2524 raft_consensus.cc:491] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.349503 2524 raft_consensus.cc:513] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.349589 1184 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.350126 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:27.350406 1184 ts_tablet_manager.cc:1397] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
W20250110 02:51:27.350744 853 leader_election.cc:343] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:27.351251 2524 leader_election.cc:290] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.351467 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4af23720048fe98b3231ba402d45c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.351656 1056 raft_consensus.cc:2463] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.351974 853 leader_election.cc:304] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.351792 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:27.352334 2524 raft_consensus.cc:2798] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.352413 2524 raft_consensus.cc:491] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.352499 2524 raft_consensus.cc:3054] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.353299 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:27.354041 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:27.354573 2224 ts_tablet_manager.cc:1918] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.354699 2224 log.cc:1198] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:27.354794 2524 raft_consensus.cc:513] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.355146 2224 ts_tablet_manager.cc:1939] T 32dd1e726f1a41d0894737b3ef3caf8f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.354602 1184 raft_consensus.cc:357] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.355274 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:27.355746 1184 raft_consensus.cc:383] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:27.355964 853 leader_election.cc:343] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:27.356262 1181 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.356441 1181 ts_tablet_manager.cc:1397] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.002s
I20250110 02:51:27.355973 1184 raft_consensus.cc:738] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.356570 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4af23720048fe98b3231ba402d45c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.356737 1056 raft_consensus.cc:3054] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.357218 1181 raft_consensus.cc:357] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.357386 1181 raft_consensus.cc:383] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.357440 1181 raft_consensus.cc:738] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.357658 1181 consensus_queue.cc:260] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.357297 1184 consensus_queue.cc:260] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.358279 1181 ts_tablet_manager.cc:1428] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:27.359313 2524 leader_election.cc:290] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.359164 1056 raft_consensus.cc:2463] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.359004 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:27.359776 853 leader_election.cc:304] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.360141 2524 raft_consensus.cc:2798] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.360368 2524 raft_consensus.cc:695] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.360481 1184 ts_tablet_manager.cc:1428] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:27.360582 2524 consensus_queue.cc:237] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.360617 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:27.361745 1184 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.364346 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 134) took 3859 ms. Trace:
I20250110 02:51:27.364487 2352 rpcz_store.cc:276] 0110 02:51:24.263209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.263900 (+ 691us) service_pool.cc:225] Handling call
0110 02:51:27.164186 (+2900286us) ts_tablet_manager.cc:1145] Deleting tablet 2e25f9643ac0476a90c93fb9aeb921a3
0110 02:51:27.164195 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.272853 (+108658us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.328472 (+ 55619us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.344343 (+ 15871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2900144}
I20250110 02:51:27.364657 2224 rpcz_store.cc:276] 0110 02:51:23.505243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.505779 (+ 536us) service_pool.cc:225] Handling call
0110 02:51:27.311432 (+3805653us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:27.311440 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.321303 (+ 9863us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.354479 (+ 33176us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.364331 (+ 9852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3805309}
I20250110 02:51:27.365017 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 2e25f9643ac0476a90c93fb9aeb921a3 (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]) successfully deleted
I20250110 02:51:27.365087 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 148) took 3101 ms. Trace:
I20250110 02:51:27.365195 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 32dd1e726f1a41d0894737b3ef3caf8f (table default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]) successfully deleted
I20250110 02:51:27.365162 2352 rpcz_store.cc:276] 0110 02:51:24.263324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264073 (+ 749us) service_pool.cc:225] Handling call
0110 02:51:27.364926 (+3100853us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:27.364938 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.365070 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3100727}
I20250110 02:51:27.364614 1181 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:27.365577 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
I20250110 02:51:27.365931 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:27.366545 2352 tablet_replica.cc:331] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:27.366751 2352 raft_consensus.cc:2238] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.366930 2352 raft_consensus.cc:2267] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.367005 1181 tablet_bootstrap.cc:654] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.367173 2224 tablet_replica.cc:331] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.367846 2352 ts_tablet_manager.cc:1905] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.368062 2224 raft_consensus.cc:2238] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.368371 2224 raft_consensus.cc:2267] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.368834 2224 ts_tablet_manager.cc:1905] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.368983 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:27.371428 1184 tablet_bootstrap.cc:654] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.371982 2357 ts_tablet_manager.cc:1918] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.374014 2357 log.cc:1198] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f5e88d1bbc304b4ebbc849a7d783052e
I20250110 02:51:27.376160 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:27.376946 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:27.378995 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:27.380000 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:27.380450 1181 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.380620 1181 ts_tablet_manager.cc:1397] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250110 02:51:27.380581 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:27.382876 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:27.384104 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43]). Expected version 3 got 0
I20250110 02:51:27.385385 2632 raft_consensus.cc:491] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:27.385504 2632 raft_consensus.cc:513] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.386121 2632 leader_election.cc:290] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.386487 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:27.385166 1181 raft_consensus.cc:357] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.386973 709 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:27.387035 1181 raft_consensus.cc:383] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.387143 1181 raft_consensus.cc:738] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.387413 1181 consensus_queue.cc:260] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.387501 711 leader_election.cc:343] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:27.387579 711 leader_election.cc:304] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.387908 2632 raft_consensus.cc:2743] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:27.388067 1181 ts_tablet_manager.cc:1428] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250110 02:51:27.388376 1181 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.389498 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:27.390544 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:27.391430 1181 tablet_bootstrap.cc:654] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.392235 2524 raft_consensus.cc:491] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.392379 1184 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.392395 2524 raft_consensus.cc:513] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.392611 994 consensus_peers.cc:487] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.392697 1184 ts_tablet_manager.cc:1397] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.002s
I20250110 02:51:27.392911 2676 tablet.cc:1721] T abbd43638a2c47dcba4f6f90e8676f42 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:27.393277 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ce9f6a3bb4e4a8d80ca0b047500a5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.393457 1056 raft_consensus.cc:2463] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.393817 853 leader_election.cc:304] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.394088 2345 raft_consensus.cc:2798] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.394179 2345 raft_consensus.cc:491] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.394274 2345 raft_consensus.cc:3054] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.394063 1184 raft_consensus.cc:357] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.394307 2524 leader_election.cc:290] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.394366 1184 raft_consensus.cc:383] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.394479 1184 raft_consensus.cc:738] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.394665 2357 ts_tablet_manager.cc:1939] T f5e88d1bbc304b4ebbc849a7d783052e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.394716 2224 ts_tablet_manager.cc:1918] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.394785 2224 log.cc:1198] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/95b14d7592024a34baf7756d92e546d9
I20250110 02:51:27.394765 1184 consensus_queue.cc:260] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.394981 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 119) took 3145 ms. Trace:
I20250110 02:51:27.395047 2357 rpcz_store.cc:276] 0110 02:51:24.249851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.249927 (+ 76us) service_pool.cc:225] Handling call
0110 02:51:27.180576 (+2930649us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:27.180584 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.333442 (+152858us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.371896 (+ 38454us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.394965 (+ 23069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2930504}
I20250110 02:51:27.395411 2357 tablet_replica.cc:331] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.396760 2345 raft_consensus.cc:513] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.397862 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f5e88d1bbc304b4ebbc849a7d783052e (table default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]) successfully deleted
I20250110 02:51:27.398391 2224 ts_tablet_manager.cc:1939] T 95b14d7592024a34baf7756d92e546d9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.398197 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:27.395680 1184 ts_tablet_manager.cc:1428] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:27.398675 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 135) took 3891 ms. Trace:
I20250110 02:51:27.398844 2357 raft_consensus.cc:2238] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.398988 2357 raft_consensus.cc:2267] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.398885 2224 rpcz_store.cc:276] 0110 02:51:23.507672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.508126 (+ 454us) service_pool.cc:225] Handling call
0110 02:51:27.367137 (+3859011us) ts_tablet_manager.cc:1145] Deleting tablet 95b14d7592024a34baf7756d92e546d9
0110 02:51:27.367146 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.376178 (+ 9032us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.394600 (+ 18422us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.398660 (+ 4060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3853771}
W20250110 02:51:27.399230 853 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:27.399472 2345 leader_election.cc:290] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.399503 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 140) took 3888 ms. Trace:
I20250110 02:51:27.399598 2224 rpcz_store.cc:276] 0110 02:51:23.511064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.513161 (+ 2097us) service_pool.cc:225] Handling call
0110 02:51:27.399393 (+3886232us) ts_tablet_manager.cc:1145] Deleting tablet f63d69d04b494a848653636c9e259105
0110 02:51:27.399402 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.399488 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3884837}
I20250110 02:51:27.400035 2224 tablet_replica.cc:331] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:27.400336 853 leader_election.cc:343] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:27.400527 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323ce9f6a3bb4e4a8d80ca0b047500a5" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.400651 1056 raft_consensus.cc:3054] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.401021 2224 raft_consensus.cc:2238] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.401226 2224 raft_consensus.cc:2267] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.401854 2224 ts_tablet_manager.cc:1905] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.402004 2352 ts_tablet_manager.cc:1918] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.402558 2357 ts_tablet_manager.cc:1905] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.399268 1184 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.402941 1056 raft_consensus.cc:2463] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.404042 1184 tablet_bootstrap.cc:654] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.408614 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 95b14d7592024a34baf7756d92e546d9 (table default.Table_bf5730ffdb414255925a1360ec7d7626 [id=8dafaf90f6e7460284b6fc3434eed1e4]) successfully deleted
I20250110 02:51:27.403292 853 leader_election.cc:304] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.409198 2345 raft_consensus.cc:2798] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.403632 2677 tablet.cc:1721] T abbd43638a2c47dcba4f6f90e8676f42 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:27.409423 2345 raft_consensus.cc:695] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.409667 2345 consensus_queue.cc:237] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.404723 1188 tablet_bootstrap.cc:492] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:27.412072 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:27.412148 1188 ts_tablet_manager.cc:1397] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.121s user 0.001s sys 0.006s
I20250110 02:51:27.412760 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:27.412947 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:27.413071 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:27.413178 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:27.413280 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:27.413914 1184 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.414007 1184 ts_tablet_manager.cc:1397] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250110 02:51:27.414490 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:27.415526 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:27.416592 1184 raft_consensus.cc:357] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.417078 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:27.417150 1184 raft_consensus.cc:383] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.417336 1184 raft_consensus.cc:738] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.417698 1184 consensus_queue.cc:260] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.418166 1444 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:27.418673 1184 ts_tablet_manager.cc:1428] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 02:51:27.418867 2352 log.cc:1198] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/344184e1c98d422095cb254f5b2bc792
I20250110 02:51:27.419317 2352 ts_tablet_manager.cc:1939] T 344184e1c98d422095cb254f5b2bc792 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.419368 1457 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_09da00340dfa4ef59164793be294d5df [id=0692620395c14593803592e6aade6af3] alter complete (version 2)
I20250110 02:51:27.420027 1184 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.421324 1184 tablet_bootstrap.cc:654] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.422482 2224 ts_tablet_manager.cc:1918] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.422569 2224 log.cc:1198] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:27.422835 2224 ts_tablet_manager.cc:1939] T d51adff7f50c4aec94e1892a50ec36c4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.423365 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 150) took 3159 ms. Trace:
I20250110 02:51:27.423525 1181 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.423431 2352 rpcz_store.cc:276] 0110 02:51:24.263540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264224 (+ 684us) service_pool.cc:225] Handling call
0110 02:51:27.366504 (+3102280us) ts_tablet_manager.cc:1145] Deleting tablet 344184e1c98d422095cb254f5b2bc792
0110 02:51:27.366513 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.399405 (+ 32892us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.401903 (+ 2498us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.423349 (+ 21446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3102146}
I20250110 02:51:27.423617 1181 ts_tablet_manager.cc:1397] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20250110 02:51:27.424263 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 144) took 3161 ms. Trace:
I20250110 02:51:27.424320 2352 rpcz_store.cc:276] 0110 02:51:24.262867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264386 (+ 1519us) service_pool.cc:225] Handling call
0110 02:51:27.424191 (+3159805us) ts_tablet_manager.cc:1145] Deleting tablet 35ec1dba54a247d5ba9f9e053aa3b645
0110 02:51:27.424199 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.424250 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3159664}
I20250110 02:51:27.426138 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 151) took 3162 ms. Trace:
I20250110 02:51:27.426213 2352 rpcz_store.cc:276] 0110 02:51:24.263665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264554 (+ 889us) service_pool.cc:225] Handling call
0110 02:51:27.426042 (+3161488us) ts_tablet_manager.cc:1145] Deleting tablet f63d69d04b494a848653636c9e259105
0110 02:51:27.426050 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.426124 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3161372}
I20250110 02:51:27.426968 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 162) took 3158 ms. Trace:
I20250110 02:51:27.427023 2352 rpcz_store.cc:276] 0110 02:51:24.268573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.268970 (+ 397us) service_pool.cc:225] Handling call
0110 02:51:27.426892 (+3157922us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:27.426899 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.426956 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3157781}
I20250110 02:51:27.427644 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 163) took 3159 ms. Trace:
I20250110 02:51:27.427702 2352 rpcz_store.cc:276] 0110 02:51:24.268645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269151 (+ 506us) service_pool.cc:225] Handling call
0110 02:51:27.427572 (+3158421us) ts_tablet_manager.cc:1145] Deleting tablet 4aa78a9ef5eb443da0c8983a7190440b
0110 02:51:27.427579 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.427632 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3158253}
I20250110 02:51:27.433384 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 344184e1c98d422095cb254f5b2bc792 (table default.Table_196d49d7fc844b019f3a7356659f5b72 [id=141731490a99451e9f722fcc3d5857bd]) successfully deleted
W20250110 02:51:27.433605 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 35ec1dba54a247d5ba9f9e053aa3b645 because the tablet was not found. No further retry: Not found: Tablet not found: 35ec1dba54a247d5ba9f9e053aa3b645
W20250110 02:51:27.433854 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51adff7f50c4aec94e1892a50ec36c4 already in progress: creating tablet
W20250110 02:51:27.434070 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:27.419072 2549 raft_consensus.cc:491] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.434827 2549 raft_consensus.cc:513] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.436754 2632 raft_consensus.cc:491] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.439337 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.439529 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.439668 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af3d61366b2432db53b18f2b7bb49d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.439774 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.439949 1035 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440021 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440060 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440208 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440335 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440472 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.440573 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.436882 2632 raft_consensus.cc:513] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.445822 2377 consensus_queue.cc:1035] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.446609 2549 leader_election.cc:290] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.449602 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.449689 2632 leader_election.cc:290] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.450055 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.450246 1056 raft_consensus.cc:2463] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:27.452775 2357 ts_tablet_manager.cc:1918] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.452901 2357 log.cc:1198] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:27.453238 2357 ts_tablet_manager.cc:1939] T 32dd1e726f1a41d0894737b3ef3caf8f P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.453608 2632 raft_consensus.cc:491] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.453733 2632 raft_consensus.cc:513] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.454249 2632 leader_election.cc:290] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.454727 2352 tablet_replica.cc:331] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:27.455307 711 leader_election.cc:304] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.455542 2632 raft_consensus.cc:2798] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.455602 2632 raft_consensus.cc:491] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.455652 2632 raft_consensus.cc:3054] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.455746 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.455842 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.455906 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.455996 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456024 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456130 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456233 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456370 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456480 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456496 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127de45264f4660b843734a530eb310 with delete_type TABLET_DATA_DELETED (replaced by tablet 8eb9ead3175140028e584e1ccb4874de) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456128 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456684 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.456954 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet dad0d561a8ef498ba13bc813d4cd82fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457123 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d10ace37b96844328ccdb821dad3c5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457227 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457263 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457372 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457401 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457563 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457693 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.457712 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f66603a0ee8c4082807c5a6aabe7e339 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.458122 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet da8f247c862e4cb68452e230857ee6b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.458527 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.459033 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f01daf4c67462ba75e1d0cf56cef12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.459461 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.459782 2632 raft_consensus.cc:513] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.459884 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4422eca5604d149e73c2a669d07af0 with delete_type TABLET_DATA_DELETED (replaced by tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.460367 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.460717 2632 leader_election.cc:290] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.460767 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.461247 2352 raft_consensus.cc:2238] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.461300 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.461805 2352 raft_consensus.cc:2267] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.462028 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.462338 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.462726 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.462805 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.462891 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463028 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463073 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463188 1184 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.463181 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463038 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463359 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463624 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.463697 2352 ts_tablet_manager.cc:1905] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.463857 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.464316 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.464491 1053 raft_consensus.cc:3054] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.464465 1443 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_ee8b07a158764473a98e0582c12d2075 [id=1afd8993888840da819f652f440b1b3e] alter complete (version 1)
I20250110 02:51:27.464705 1184 ts_tablet_manager.cc:1397] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250110 02:51:27.467487 1053 raft_consensus.cc:2463] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:27.468263 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.468470 1184 raft_consensus.cc:357] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.469792 2524 consensus_queue.cc:1035] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.474699 2378 raft_consensus.cc:491] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:27.475591 853 consensus_peers.cc:487] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.472682 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
W20250110 02:51:27.473541 711 leader_election.cc:343] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.479193 751 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.479467 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.485528 711 leader_election.cc:304] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.485586 746 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.479880 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.480371 744 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.480461 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.480994 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.481156 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.481277 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.481853 750 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.482227 749 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.486326 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.486569 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.488189 2632 raft_consensus.cc:2798] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.488572 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b375a932156d4abb908dbf327689b0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 19e4af23720048fe98b3231ba402d45c) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.488724 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 76833ce515cf4ee9bc8fff0f626af220 with delete_type TABLET_DATA_DELETED (replaced by tablet b74beb77505f472fb6b9e4a30bfe6582) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.489145 2632 raft_consensus.cc:695] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
W20250110 02:51:27.489595 711 leader_election.cc:343] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.489413 2632 consensus_queue.cc:237] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.490469 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 28340f0f9a2043a199c74f13755eb702 with delete_type TABLET_DATA_DELETED (replaced by tablet 19b90cc15dbf47d4b4dbaf12d588abb0) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.474798 2378 raft_consensus.cc:513] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.494616 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d9018955f3884824beb2e7336987fc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.494681 745 tablet_service.cc:1514] Processing DeleteTablet for tablet d20177229f7b4f11875e0007afb1dd00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:27.496130 853 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:27.496469 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b382d634db9642df8e00c5ccaa5787a9" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.496647 1061 raft_consensus.cc:2463] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.497668 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:27.496960 853 leader_election.cc:304] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.494400 2378 leader_election.cc:290] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.499512 2378 raft_consensus.cc:2798] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.499590 2378 raft_consensus.cc:491] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.499660 2378 raft_consensus.cc:3054] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.500187 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.503763 1440 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 118) took 3849 ms (client timeout 1000 ms). Trace:
W20250110 02:51:27.508867 1450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84adc1fd3f76497b98a0fcbc52a3120a consensus state has no leader
I20250110 02:51:27.509096 1450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 105) took 2783 ms. Trace:
I20250110 02:51:27.509220 1450 rpcz_store.cc:276] 0110 02:51:24.725212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.135168 (+2409956us) service_pool.cc:225] Handling call
0110 02:51:27.135498 (+ 330us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.135725 (+ 227us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:27.135880 (+ 155us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:27.135941 (+ 61us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:27.147563 (+ 11622us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:27.509025 (+361462us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.137138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.137277 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.137282 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.137284 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.137423 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.137434 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.137439 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.137514 (+ 75us) tablet.cc:698] Row locks acquired
0110 02:51:27.137569 (+ 55us) write_op.cc:260] PREPARE: finished
0110 02:51:27.137624 (+ 55us) write_op.cc:270] Start()
0110 02:51:27.137685 (+ 61us) write_op.cc:276] Timestamp: P: 1736477487137574 usec, L: 0
0110 02:51:27.137691 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.137897 (+ 206us) log.cc:844] Serialized 405 byte log entry
0110 02:51:27.144747 (+ 6850us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.147193 (+ 2446us) write_op.cc:301] APPLY: starting
0110 02:51:27.147210 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.147245 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.147272 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.147313 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.147340 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.147345 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:27.147413 (+ 68us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.147477 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.147492 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:27.147497 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.147643 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.147647 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2394,"apply.run_cpu_time_us":369,"apply.run_wall_time_us":425,"num_ops":1,"prepare.queue_time_us":748,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":1853,"raft.queue_time_us":49,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":381,"replication_time_us":7030}]]}
I20250110 02:51:27.512895 2600 raft_consensus.cc:491] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.513029 2600 raft_consensus.cc:513] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.513655 2600 leader_election.cc:290] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:27.515261 996 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:27.518931 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 136) took 4008 ms. Trace:
I20250110 02:51:27.519013 2224 rpcz_store.cc:276] 0110 02:51:23.510759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.513221 (+ 2462us) service_pool.cc:225] Handling call
0110 02:51:27.399979 (+3886758us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:27.399986 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.419989 (+ 20003us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.422411 (+ 2422us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.518913 (+ 96502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3885294}
I20250110 02:51:27.519181 2600 consensus_queue.cc:1035] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.519488 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 138) took 4008 ms. Trace:
I20250110 02:51:27.519553 2224 rpcz_store.cc:276] 0110 02:51:23.510910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.513067 (+ 2157us) service_pool.cc:225] Handling call
0110 02:51:27.519386 (+4006319us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:27.519396 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.519474 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4004583}
I20250110 02:51:27.520022 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 137) took 4009 ms. Trace:
I20250110 02:51:27.520112 2224 rpcz_store.cc:276] 0110 02:51:23.510846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.513196 (+ 2350us) service_pool.cc:225] Handling call
0110 02:51:27.519913 (+4006717us) ts_tablet_manager.cc:1145] Deleting tablet f14f06ba7d0f44418693e36f6bc6d888
0110 02:51:27.519920 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.519988 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4004986}
I20250110 02:51:27.520432 2549 raft_consensus.cc:491] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.520553 2224 tablet_replica.cc:331] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.520565 2549 raft_consensus.cc:513] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.520817 2224 raft_consensus.cc:2238] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.521272 2224 raft_consensus.cc:2267] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.521708 2549 leader_election.cc:290] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.522338 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.522800 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d51adff7f50c4aec94e1892a50ec36c4 (table default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]) successfully deleted
W20250110 02:51:27.522867 711 leader_election.cc:343] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:27.523030 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80186dd7d1a240068584b64ce9e9b03f already in progress: creating tablet
W20250110 02:51:27.523212 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f14f06ba7d0f44418693e36f6bc6d888 because the tablet was not found. No further retry: Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888
W20250110 02:51:27.523439 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4bd2283e7ed7456dae466a8582587eb7 consensus state has no leader
I20250110 02:51:27.526559 745 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fd370138ac4039be55642f1b4c5607 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e554d00d05b4fa7a14c9b6159f9d726) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.527866 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.527902 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.528532 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.528569 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.528865 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 240b64d551a548e884ee2a7238c0363a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.529728 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.530098 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.530654 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:27.530689 1443 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet dc6b2654f86e4686b8a3034de02d11bf consensus state has no leader
I20250110 02:51:27.530859 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.530918 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.531455 745 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.531589 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.531647 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.531780 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.482503 2687 tablet.cc:1721] T abbd43638a2c47dcba4f6f90e8676f42 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:27.482908 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.483356 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.483744 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.484447 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.484570 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.484687 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:27.535274 1440 rpcz_store.cc:269] 0110 02:51:23.654758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.672944 (+ 18186us) service_pool.cc:225] Handling call
0110 02:51:27.052048 (+3379104us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.052055 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:27.132927 (+ 80872us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.132957 (+ 30us) write_op.cc:454] Released schema lock
0110 02:51:27.503686 (+370729us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:24.089079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.089186 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.089192 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:24.089195 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:24.089327 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.089340 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.089345 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:24.089447 (+ 102us) tablet.cc:698] Row locks acquired
0110 02:51:24.089451 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:24.089487 (+ 36us) write_op.cc:270] Start()
0110 02:51:24.089523 (+ 36us) write_op.cc:276] Timestamp: P: 1736477484089422 usec, L: 0
0110 02:51:24.089529 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.089698 (+ 169us) log.cc:844] Serialized 901 byte log entry
0110 02:51:24.093965 (+ 4267us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.094098 (+ 133us) write_op.cc:301] APPLY: starting
0110 02:51:24.094113 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.094161 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.094185 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.094249 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.094282 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:24.094289 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:24.094360 (+ 71us) log.cc:844] Serialized 29 byte log entry
0110 02:51:24.094437 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.094456 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:24.094462 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.094530 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.094535 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:24.205419 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:24.205513 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:24.205520 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:24.205522 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:24.205653 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:24.205665 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:24.205670 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:24.205727 (+ 57us) tablet.cc:698] Row locks acquired
0110 02:51:24.205729 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:24.205763 (+ 34us) write_op.cc:270] Start()
0110 02:51:24.205794 (+ 31us) write_op.cc:276] Timestamp: P: 1736477484205737 usec, L: 0
0110 02:51:24.205798 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:24.206001 (+ 203us) log.cc:844] Serialized 356 byte log entry
0110 02:51:24.208716 (+ 2715us) op_driver.cc:464] REPLICATION: finished
0110 02:51:24.208840 (+ 124us) write_op.cc:301] APPLY: starting
0110 02:51:24.208854 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:24.208915 (+ 61us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:24.208963 (+ 48us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:24.209019 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:24.209056 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:24.209065 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:24.209159 (+ 94us) log.cc:844] Serialized 23 byte log entry
0110 02:51:24.209227 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.209245 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:24.209251 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:24.209328 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:24.209334 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:25.056367 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.056429 (+ 62us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.056435 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:25.056438 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.056522 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.056575 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.056582 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.056637 (+ 55us) tablet.cc:698] Row locks acquired
0110 02:51:25.056641 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:25.056675 (+ 34us) write_op.cc:270] Start()
0110 02:51:25.056704 (+ 29us) write_op.cc:276] Timestamp: P: 1736477485056614 usec, L: 0
0110 02:51:25.056706 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.056871 (+ 165us) log.cc:844] Serialized 356 byte log entry
0110 02:51:25.076948 (+ 20077us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.077127 (+ 179us) write_op.cc:301] APPLY: starting
0110 02:51:25.077140 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.077187 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.077208 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.077253 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.077289 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.077299 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:25.077362 (+ 63us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.077426 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.077440 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:25.077446 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.077517 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.077520 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.026234 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.026411 (+ 177us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.026418 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:27.026421 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:27.026528 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.026540 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.026545 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.026601 (+ 56us) tablet.cc:698] Row locks acquired
0110 02:51:27.026603 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.026631 (+ 28us) write_op.cc:270] Start()
0110 02:51:27.026659 (+ 28us) write_op.cc:276] Timestamp: P: 1736477487026594 usec, L: 0
0110 02:51:27.026662 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.026802 (+ 140us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.031672 (+ 4870us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.031787 (+ 115us) write_op.cc:301] APPLY: starting
0110 02:51:27.031804 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.031846 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.031873 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.031929 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.031968 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.031978 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:27.032051 (+ 73us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.032123 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.032142 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:27.032150 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:27.118402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.118501 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.118507 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:27.118509 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.118641 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.118652 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.118657 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.118732 (+ 75us) tablet.cc:698] Row locks acquired
0110 02:51:27.118735 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.118764 (+ 29us) write_op.cc:270] Start()
0110 02:51:27.118790 (+ 26us) write_op.cc:276] Timestamp: P: 1736477487118717 usec, L: 0
0110 02:51:27.118794 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.119462 (+ 668us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.122821 (+ 3359us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.122913 (+ 92us) write_op.cc:301] APPLY: starting
0110 02:51:27.122924 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.122955 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.122976 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.123021 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.123047 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.123053 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:27.123115 (+ 62us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.123162 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.123175 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:27.123180 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":72026,"reported_tablets":269,"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":421,"apply.run_wall_time_us":421,"num_ops":2,"prepare.queue_time_us":250,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":747,"raft.queue_time_us":180,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":225,"replication_time_us":4409}],["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":476,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":713,"raft.queue_time_us":182,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":242,"replication_time_us":2891}],["op",{"apply.queue_time_us":120,"apply.run_cpu_time_us":385,"apply.run_wall_time_us":386,"num_ops":1,"prepare.queue_time_us":5227,"prepare.run_cpu_time_us":551,"prepare.run_wall_time_us":642,"raft.queue_time_us":33,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":20205}],["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":415,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":655,"raft.queue_time_us":65,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":205,"replication_time_us":4958}],["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":308,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":1697,"raft.queue_time_us":43,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"replication_time_us":4005}]]}
I20250110 02:51:27.540056 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:27.543437 1184 raft_consensus.cc:383] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.543562 1184 raft_consensus.cc:738] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.514314 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65285bce60fd4015b085deb7442eea6e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.543866 1184 consensus_queue.cc:260] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.544296 994 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.544399 1184 ts_tablet_manager.cc:1428] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20250110 02:51:27.544445 994 leader_election.cc:304] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.544705 1184 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.545092 2600 raft_consensus.cc:2743] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.546326 1184 tablet_bootstrap.cc:654] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.543639 2378 raft_consensus.cc:513] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.548010 2378 leader_election.cc:290] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:27.558197 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 61259.2us
I20250110 02:51:27.558732 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b382d634db9642df8e00c5ccaa5787a9" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.558955 1061 raft_consensus.cc:3054] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:27.560127 1443 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 127) took 2421 ms (client timeout 1000 ms). Trace:
W20250110 02:51:27.560263 1443 rpcz_store.cc:269] 0110 02:51:25.138622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.140308 (+ 1686us) service_pool.cc:225] Handling call
0110 02:51:27.560054 (+2419746us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.224451 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.224546 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.224552 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:27.224554 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.224643 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.224654 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.224658 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.224709 (+ 51us) tablet.cc:698] Row locks acquired
0110 02:51:27.224712 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.224737 (+ 25us) write_op.cc:270] Start()
0110 02:51:27.224762 (+ 25us) write_op.cc:276] Timestamp: P: 1736477487224682 usec, L: 0
0110 02:51:27.224766 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.224906 (+ 140us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.230822 (+ 5916us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.230962 (+ 140us) write_op.cc:301] APPLY: starting
0110 02:51:27.230979 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.231028 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.231092 (+ 64us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.231153 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.231194 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.231204 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:27.231283 (+ 79us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.231346 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.231368 (+ 22us) write_op.cc:454] Released schema lock
0110 02:51:27.231377 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.231451 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.231456 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":151917,"reported_tablets":251,"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":478,"apply.run_wall_time_us":478,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":542,"prepare.run_wall_time_us":541,"raft.queue_time_us":65,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"replication_time_us":6022}]]}
I20250110 02:51:27.561627 1061 raft_consensus.cc:2463] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
W20250110 02:51:27.562157 853 consensus_peers.cc:487] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:27.563192 853 consensus_peers.cc:487] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9507a5283d8f4d07a293ba474c642298. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.564046 853 leader_election.cc:304] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.564857 2692 raft_consensus.cc:2798] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.564946 2692 raft_consensus.cc:695] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:27.565142 2692 consensus_queue.cc:237] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.565990 2352 ts_tablet_manager.cc:1918] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.566082 2352 log.cc:1198] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:27.566392 2352 ts_tablet_manager.cc:1939] T 3fa76357abae4a95b8fb3af47288e43a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.566673 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 168) took 3297 ms. Trace:
I20250110 02:51:27.566741 2352 rpcz_store.cc:276] 0110 02:51:24.269233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269349 (+ 116us) service_pool.cc:225] Handling call
0110 02:51:27.454680 (+3185331us) ts_tablet_manager.cc:1145] Deleting tablet 3fa76357abae4a95b8fb3af47288e43a
0110 02:51:27.454689 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.524930 (+ 70241us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.565869 (+ 40939us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.566659 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185202}
I20250110 02:51:27.567117 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 170) took 3297 ms. Trace:
I20250110 02:51:27.567183 2352 rpcz_store.cc:276] 0110 02:51:24.269444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269502 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:27.566957 (+3297455us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:27.567026 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.567104 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3297339}
I20250110 02:51:27.567384 2352 tablet_replica.cc:331] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:27.567551 2352 raft_consensus.cc:2238] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.567775 2352 raft_consensus.cc:2267] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.573326 2224 ts_tablet_manager.cc:1905] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.574910 2352 ts_tablet_manager.cc:1905] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.595062 756 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.581562 2378 raft_consensus.cc:491] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.595378 2378 raft_consensus.cc:513] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.596076 2378 leader_election.cc:290] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.598459 2456 consensus_queue.cc:1035] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:27.598897 996 consensus_peers.cc:487] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.599272 2378 raft_consensus.cc:491] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.599401 2378 raft_consensus.cc:513] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.582228 2377 consensus_queue.cc:1035] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.600003 2378 leader_election.cc:290] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.601148 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de9cf980d9c4117b4ec3ca937cfcf01" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.601332 1061 raft_consensus.cc:2463] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.601334 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.601634 853 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:27.602178 853 leader_election.cc:304] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.602475 2377 raft_consensus.cc:2798] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.602536 2377 raft_consensus.cc:491] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.602586 2377 raft_consensus.cc:3054] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.604954 2377 raft_consensus.cc:513] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.605479 2377 leader_election.cc:290] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.605778 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.606352 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de9cf980d9c4117b4ec3ca937cfcf01" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.606527 1061 raft_consensus.cc:3054] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.606676 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:27.607761 853 leader_election.cc:343] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:27.608444 2549 raft_consensus.cc:491] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.608722 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.608862 1061 raft_consensus.cc:2463] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:27.609174 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:27.609508 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.609699 853 leader_election.cc:304] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:27.557771 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 120) took 3307 ms. Trace:
I20250110 02:51:27.610440 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:27.610625 709 leader_election.cc:343] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.611174 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.611356 748 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.611464 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.611671 770 raft_consensus.cc:2463] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:27.612102 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.612062 2549 raft_consensus.cc:513] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.612713 2549 leader_election.cc:290] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.617259 2378 raft_consensus.cc:2798] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:27.617309 853 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:27.617368 2378 raft_consensus.cc:695] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
W20250110 02:51:27.617609 853 leader_election.cc:343] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:27.617554 2378 consensus_queue.cc:237] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.617919 853 leader_election.cc:304] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0; no voters: 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.618783 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.619678 711 leader_election.cc:343] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.622429 2357 rpcz_store.cc:276] 0110 02:51:24.249938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250001 (+ 63us) service_pool.cc:225] Handling call
0110 02:51:27.395368 (+3145367us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:27.395376 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.408162 (+ 12786us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.452682 (+ 44520us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.557751 (+105069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3145128}
I20250110 02:51:27.611310 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 3fa76357abae4a95b8fb3af47288e43a (table default.Table_dce75d4fa88e4bf59fe18cf4725d953f [id=7c8bb5b8a91a4ed9a81403e2e6196406]) successfully deleted
I20250110 02:51:27.622997 2357 tablet_replica.cc:331] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.622934 1443 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_99ac393aee99471a8a4a750750f72cc8" } modify_external_catalogs: true
I20250110 02:51:27.623158 1443 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_99ac393aee99471a8a4a750750f72cc8" } modify_external_catalogs: true
I20250110 02:51:27.623209 2357 raft_consensus.cc:2238] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.623350 2357 raft_consensus.cc:2267] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.623782 2357 ts_tablet_manager.cc:1905] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.624940 2689 raft_consensus.cc:2798] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.625028 2689 raft_consensus.cc:491] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.625084 2689 raft_consensus.cc:3054] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:27.631202 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb726d2c9a04963b92a3fb625cf84f3 already in progress: creating tablet
I20250110 02:51:27.631263 2689 raft_consensus.cc:513] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.631352 2224 ts_tablet_manager.cc:1918] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.631448 2224 log.cc:1198] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:27.631851 2689 leader_election.cc:290] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.632148 2224 ts_tablet_manager.cc:1939] T e744cdc1fcb94d46b48b684763ecf1bf P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.632597 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 141) took 4065 ms. Trace:
I20250110 02:51:27.633054 2224 rpcz_store.cc:276] 0110 02:51:23.567552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.567624 (+ 72us) service_pool.cc:225] Handling call
0110 02:51:27.520492 (+3952868us) ts_tablet_manager.cc:1145] Deleting tablet e744cdc1fcb94d46b48b684763ecf1bf
0110 02:51:27.520501 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.576693 (+ 56192us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.631254 (+ 54561us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.632561 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952569}
I20250110 02:51:27.633795 2224 tablet_replica.cc:331] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.634232 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4d3270e3a04cc99f90943b9573ec7e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:27.635239 853 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Illegal state: Tablet not RUNNING: STOPPED
W20250110 02:51:27.635908 853 leader_election.cc:343] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:27.637583 853 leader_election.cc:304] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:27.638280 1456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 126) took 3515 ms (client timeout 1000 ms). Trace:
W20250110 02:51:27.638433 1456 rpcz_store.cc:269] 0110 02:51:24.122712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.155605 (+ 32893us) service_pool.cc:225] Handling call
0110 02:51:27.638212 (+3482607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.061727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.061818 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.061823 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.061825 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.118444 (+ 56619us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.118597 (+ 153us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.118604 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.118714 (+ 110us) tablet.cc:698] Row locks acquired
0110 02:51:25.118717 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.119358 (+ 641us) write_op.cc:270] Start()
0110 02:51:25.119551 (+ 193us) write_op.cc:276] Timestamp: P: 1736477485119278 usec, L: 0
0110 02:51:25.119557 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.133171 (+ 13614us) log.cc:844] Serialized 356 byte log entry
0110 02:51:25.136427 (+ 3256us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:25.163629 (+ 27202us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.163780 (+ 151us) write_op.cc:301] APPLY: starting
0110 02:51:25.163799 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.163847 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.163849 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.163906 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.163936 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.163942 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.164013 (+ 71us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.164092 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.164110 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:25.164116 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.164183 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.164187 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":329662,"reported_tablets":255,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":396,"num_ops":1,"prepare.queue_time_us":2038,"prepare.run_cpu_time_us":1276,"prepare.run_wall_time_us":74754,"raft.queue_time_us":676,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":322,"replication_time_us":44039,"spinlock_wait_cycles":30857472}]]}
I20250110 02:51:27.639598 1456 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e8505c78e8614d1b9424cf824a59dbb9" } modify_external_catalogs: true
I20250110 02:51:27.639853 1456 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e8505c78e8614d1b9424cf824a59dbb9" } modify_external_catalogs: true
I20250110 02:51:27.641412 2345 raft_consensus.cc:2743] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.582298 1181 raft_consensus.cc:357] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.646557 1181 raft_consensus.cc:383] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.646620 1181 raft_consensus.cc:738] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.646822 1181 consensus_queue.cc:260] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.647241 1181 ts_tablet_manager.cc:1428] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.065s user 0.004s sys 0.000s
I20250110 02:51:27.647481 1181 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.648798 1181 tablet_bootstrap.cc:654] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.624665 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 32dd1e726f1a41d0894737b3ef3caf8f (table default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]) successfully deleted
I20250110 02:51:27.652338 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e744cdc1fcb94d46b48b684763ecf1bf (table default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]) successfully deleted
I20250110 02:51:27.655275 2357 ts_tablet_manager.cc:1918] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.655416 2357 log.cc:1198] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:27.655689 2357 ts_tablet_manager.cc:1939] T 6e4d3270e3a04cc99f90943b9573ec7e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.657956 2224 raft_consensus.cc:2238] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.658186 2224 raft_consensus.cc:2267] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.661146 2224 ts_tablet_manager.cc:1905] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.675302 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3ce95413df40369ee599176f91b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.681007 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:27.681427 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:27.682456 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:27.682698 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:27.683102 1184 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.683149 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:27.683235 1184 ts_tablet_manager.cc:1397] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.003s
I20250110 02:51:27.683545 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:27.683743 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:27.683945 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:27.684206 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:27.684412 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:27.684687 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:27.684950 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:27.685626 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
W20250110 02:51:27.688385 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 77767.3us
I20250110 02:51:27.688519 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:27.689173 1450 catalog_manager.cc:5526] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:27.689949 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:27.690603 1450 catalog_manager.cc:5526] T 5a9d524fda0d4848b333df9e20a9f68b P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:27.691107 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:27.691991 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:27.692615 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:27.698168 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.702857 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.702944 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e183122d13a4c38988bf2fe6f7a3694 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.703536 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.703845 881 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.704128 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.704419 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.704692 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a7397e41f7374e9fb1621dbcf30af703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.704974 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.705263 881 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.705540 881 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.705839 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e534bf66f044b88d8cda0f3a1228f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:27.706665 1448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 128) took 1427 ms (client timeout 1000 ms). Trace:
W20250110 02:51:27.706820 1448 rpcz_store.cc:269] 0110 02:51:26.278941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.342741 (+ 63800us) service_pool.cc:225] Handling call
0110 02:51:27.706612 (+1363871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":132324,"reported_tablets":241}
I20250110 02:51:27.707314 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_a2d5d1bf3c504747b630f2dd924d83f7" } modify_external_catalogs: true
I20250110 02:51:27.707541 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_a2d5d1bf3c504747b630f2dd924d83f7" } modify_external_catalogs: true
I20250110 02:51:27.708096 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:27.709034 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:27.710351 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.711416 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.711483 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:27.714457 963 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:27.715420 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:27.716343 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
W20250110 02:51:27.715967 1441 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 120) took 1590 ms (client timeout 1000 ms). Trace:
I20250110 02:51:27.716882 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:27.717303 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:27.718878 881 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:27.719106 1449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 119) took 2818 ms (client timeout 1000 ms). Trace:
I20250110 02:51:27.719202 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.719523 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef72e05f8ed4d46b759c7a04497653b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.719894 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.720539 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 344184e1c98d422095cb254f5b2bc792 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.720813 1181 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.720873 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.721149 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.721153 1181 ts_tablet_manager.cc:1397] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20250110 02:51:27.721443 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa76357abae4a95b8fb3af47288e43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.721697 881 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.722398 1181 raft_consensus.cc:357] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.722718 1181 raft_consensus.cc:383] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.722851 1181 raft_consensus.cc:738] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:27.716679 1441 rpcz_store.cc:269] 0110 02:51:26.125166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.303392 (+178226us) service_pool.cc:225] Handling call
0110 02:51:26.661851 (+358459us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.661860 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:27.042560 (+380700us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.042564 (+ 4us) write_op.cc:454] Released schema lock
0110 02:51:27.151905 (+109341us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.151910 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:27.715924 (+564014us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:26.614174 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:26.614364 (+ 190us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:26.614388 (+ 24us) write_op.cc:435] Acquired schema lock
0110 02:51:26.614391 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:26.614490 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:26.614501 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:26.614505 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:26.614555 (+ 50us) tablet.cc:698] Row locks acquired
0110 02:51:26.614556 (+ 1us) write_op.cc:260] PREPARE: finished
0110 02:51:26.614593 (+ 37us) write_op.cc:270] Start()
0110 02:51:26.614624 (+ 31us) write_op.cc:276] Timestamp: P: 1736477486614551 usec, L: 0
0110 02:51:26.614629 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:26.614809 (+ 180us) log.cc:844] Serialized 527 byte log entry
0110 02:51:26.617802 (+ 2993us) op_driver.cc:464] REPLICATION: finished
0110 02:51:26.617919 (+ 117us) write_op.cc:301] APPLY: starting
0110 02:51:26.617951 (+ 32us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:26.618008 (+ 57us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:26.618040 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:26.618084 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:26.618121 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:26.618127 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:26.618196 (+ 69us) log.cc:844] Serialized 23 byte log entry
0110 02:51:26.618267 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.618290 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:26.618299 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:26.984522 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:26.984629 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:26.984635 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:26.984637 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:26.984722 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:26.984733 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:26.984737 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:26.984787 (+ 50us) tablet.cc:698] Row locks acquired
0110 02:51:26.984788 (+ 1us) write_op.cc:260] PREPARE: finished
0110 02:51:26.984815 (+ 27us) write_op.cc:270] Start()
0110 02:51:26.984842 (+ 27us) write_op.cc:276] Timestamp: P: 1736477486984783 usec, L: 0
0110 02:51:26.984847 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:26.985028 (+ 181us) log.cc:844] Serialized 356 byte log entry
0110 02:51:26.988921 (+ 3893us) op_driver.cc:464] REPLICATION: finished
0110 02:51:26.989035 (+ 114us) write_op.cc:301] APPLY: starting
0110 02:51:26.989049 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:26.989083 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:26.989126 (+ 43us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:26.989170 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:26.989213 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:26.989219 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:26.989280 (+ 61us) log.cc:844] Serialized 23 byte log entry
0110 02:51:26.989348 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.989365 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:26.989370 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:26.989437 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.989440 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.035002 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.035106 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.035110 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:27.035112 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.035194 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.035203 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.035207 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.035249 (+ 42us) tablet.cc:698] Row locks acquired
0110 02:51:27.035251 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.035277 (+ 26us) write_op.cc:270] Start()
0110 02:51:27.035298 (+ 21us) write_op.cc:276] Timestamp: P: 1736477487035240 usec, L: 0
0110 02:51:27.035301 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.035420 (+ 119us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.038785 (+ 3365us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.038874 (+ 89us) write_op.cc:301] APPLY: starting
0110 02:51:27.038887 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.038919 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.038954 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.038990 (+ 36us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.039026 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.039038 (+ 12us) write_op.cc:312] APPLY: finished
0110 02:51:27.039158 (+ 120us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.039222 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.039238 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:27.039245 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:27.081639 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.081733 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.081738 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.081740 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.081840 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.081852 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.081856 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.081918 (+ 62us) tablet.cc:698] Row locks acquired
0110 02:51:27.081921 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.081949 (+ 28us) write_op.cc:270] Start()
0110 02:51:27.082006 (+ 57us) write_op.cc:276] Timestamp: P: 1736477487081906 usec, L: 0
0110 02:51:27.082010 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.082180 (+ 170us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.085105 (+ 2925us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.085222 (+ 117us) write_op.cc:301] APPLY: starting
0110 02:51:27.085237 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.085277 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.085351 (+ 74us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.085404 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.085441 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.085450 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:27.085520 (+ 70us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.085579 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.085598 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:27.085605 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.085678 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.085683 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.104069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.104197 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.104203 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:27.104205 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.104937 (+ 732us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.104955 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.104962 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.105058 (+ 96us) tablet.cc:698] Row locks acquired
0110 02:51:27.105063 (+ 5us) write_op.cc:260] PREPARE: finished
0110 02:51:27.105113 (+ 50us) write_op.cc:270] Start()
0110 02:51:27.105153 (+ 40us) write_op.cc:276] Timestamp: P: 1736477487105064 usec, L: 0
0110 02:51:27.105159 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.105354 (+ 195us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.109001 (+ 3647us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.109153 (+ 152us) write_op.cc:301] APPLY: starting
0110 02:51:27.109184 (+ 31us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.109235 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.109275 (+ 40us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.109337 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.109379 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.109388 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:27.109473 (+ 85us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.109539 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.109562 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:27.109570 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":132578,"reported_tablets":264,"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":446,"num_ops":1,"prepare.queue_time_us":8409,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":727,"raft.queue_time_us":376,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":204,"replication_time_us":3146}],["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":395,"apply.run_wall_time_us":395,"num_ops":1,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":609,"raft.queue_time_us":71,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":186,"replication_time_us":4045}],["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":432,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":494,"prepare.run_wall_time_us":494,"raft.queue_time_us":54,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"replication_time_us":3457}],["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":445,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":624,"prepare.run_wall_time_us":624,"raft.queue_time_us":56,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"replication_time_us":3052}],["op",{"apply.queue_time_us":93,"apply.run_cpu_time_us":479,"apply.run_wall_time_us":478,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":802,"prepare.run_wall_time_us":1718,"raft.queue_time_us":43,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":256,"replication_time_us":3805}]]}
I20250110 02:51:27.723237 1181 consensus_queue.cc:260] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.725266 1441 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_01904216609c46c581a0b70981481bb9" } modify_external_catalogs: true
I20250110 02:51:27.725457 1441 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_01904216609c46c581a0b70981481bb9" } modify_external_catalogs: true
I20250110 02:51:27.726146 1181 ts_tablet_manager.cc:1428] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250110 02:51:27.725845 1449 rpcz_store.cc:269] 0110 02:51:24.901001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.957581 (+ 56580us) service_pool.cc:225] Handling call
0110 02:51:26.948283 (+1990702us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.948289 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:27.719054 (+770765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.363797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.363879 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.363884 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:25.363887 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:25.363978 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.363985 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.363989 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:25.364075 (+ 86us) tablet.cc:698] Row locks acquired
0110 02:51:25.364078 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.364105 (+ 27us) write_op.cc:270] Start()
0110 02:51:25.364128 (+ 23us) write_op.cc:276] Timestamp: P: 1736477485364070 usec, L: 0
0110 02:51:25.364131 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.364282 (+ 151us) log.cc:844] Serialized 901 byte log entry
0110 02:51:25.368693 (+ 4411us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.368793 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:25.368822 (+ 29us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.368854 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.368876 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.368938 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.368966 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:25.368972 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.369034 (+ 62us) log.cc:844] Serialized 29 byte log entry
0110 02:51:25.369077 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.369091 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:25.369095 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.369151 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.369155 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:25.387262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.387353 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.387357 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:25.387359 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.387431 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.387438 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.387442 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.387486 (+ 44us) tablet.cc:698] Row locks acquired
0110 02:51:25.387489 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.387517 (+ 28us) write_op.cc:270] Start()
0110 02:51:25.387538 (+ 21us) write_op.cc:276] Timestamp: P: 1736477485387480 usec, L: 0
0110 02:51:25.387541 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.387653 (+ 112us) log.cc:844] Serialized 356 byte log entry
0110 02:51:25.403608 (+ 15955us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.403754 (+ 146us) write_op.cc:301] APPLY: starting
0110 02:51:25.403768 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.403812 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.403846 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.403891 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.403919 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.403925 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:25.403994 (+ 69us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.404067 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.404082 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:25.404093 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:25.404164 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.404168 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:26.942330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:26.942452 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:26.942459 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:26.942461 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:26.942556 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:26.942569 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:26.942573 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:26.942635 (+ 62us) tablet.cc:698] Row locks acquired
0110 02:51:26.942637 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:26.942666 (+ 29us) write_op.cc:270] Start()
0110 02:51:26.942698 (+ 32us) write_op.cc:276] Timestamp: P: 1736477486942637 usec, L: 0
0110 02:51:26.942701 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:26.942868 (+ 167us) log.cc:844] Serialized 356 byte log entry
0110 02:51:26.945880 (+ 3012us) op_driver.cc:464] REPLICATION: finished
0110 02:51:26.945993 (+ 113us) write_op.cc:301] APPLY: starting
0110 02:51:26.946009 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:26.946048 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:26.946096 (+ 48us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:26.946145 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:26.946178 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:26.946185 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:26.946257 (+ 72us) log.cc:844] Serialized 23 byte log entry
0110 02:51:26.946312 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.946331 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:26.946338 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:26.956229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:26.956325 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:26.956343 (+ 18us) write_op.cc:435] Acquired schema lock
0110 02:51:26.956345 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:26.956452 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:26.956463 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:26.956467 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:26.956521 (+ 54us) tablet.cc:698] Row locks acquired
0110 02:51:26.956523 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:26.956550 (+ 27us) write_op.cc:270] Start()
0110 02:51:26.956577 (+ 27us) write_op.cc:276] Timestamp: P: 1736477486956520 usec, L: 0
0110 02:51:26.956580 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:26.956752 (+ 172us) log.cc:844] Serialized 356 byte log entry
0110 02:51:26.960232 (+ 3480us) op_driver.cc:464] REPLICATION: finished
0110 02:51:26.960316 (+ 84us) write_op.cc:301] APPLY: starting
0110 02:51:26.960329 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:26.960362 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:26.964121 (+ 3759us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:26.964183 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:26.964217 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:26.964227 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:26.964304 (+ 77us) log.cc:844] Serialized 23 byte log entry
0110 02:51:26.964371 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.964390 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:26.964397 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:26.964468 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.964472 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":144017,"reported_tablets":273,"child_traces":[["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":351,"apply.run_wall_time_us":351,"mutex_wait_us":29998,"num_ops":2,"prepare.queue_time_us":30046,"prepare.run_cpu_time_us":549,"prepare.run_wall_time_us":547,"replication_time_us":4535}],["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":400,"num_ops":1,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":431,"prepare.run_wall_time_us":430,"replication_time_us":16040}],["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":424,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":625,"prepare.run_wall_time_us":625,"raft.queue_time_us":376,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":184,"replication_time_us":3150}],["op",{"apply.queue_time_us":45,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":4140,"num_ops":1,"prepare.queue_time_us":6257,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":609,"raft.queue_time_us":53,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":201,"replication_time_us":3628}]]}
I20250110 02:51:27.727679 2352 ts_tablet_manager.cc:1918] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.727783 2352 log.cc:1198] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/9507a5283d8f4d07a293ba474c642298
I20250110 02:51:27.728163 2352 ts_tablet_manager.cc:1939] T 9507a5283d8f4d07a293ba474c642298 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.726999 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:27.729830 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:27.730168 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:27.730342 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:27.730500 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:27.730644 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:27.730836 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:27.731217 1450 catalog_manager.cc:5526] T fb2272dfd0b14113afa3e13b38de6f8c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:27.731696 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:27.735491 1449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_8823b1f4fcd7469c8d4eb6264e068b2f" } modify_external_catalogs: true
I20250110 02:51:27.735723 1449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_8823b1f4fcd7469c8d4eb6264e068b2f" } modify_external_catalogs: true
W20250110 02:51:27.721238 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 108) took 2074 ms (client timeout 1000 ms). Trace:
I20250110 02:51:27.738251 1188 raft_consensus.cc:357] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.734867 2456 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85f9597077934f1a9814d284ed06d13d)
I20250110 02:51:27.738538 2456 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.739359 1188 raft_consensus.cc:383] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.740101 2456 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.741143 1181 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:27.736799 1457 rpcz_store.cc:269] 0110 02:51:25.647143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.818787 (+171644us) service_pool.cc:225] Handling call
0110 02:51:27.135585 (+1316798us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.135591 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:27.721187 (+585596us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:26.327955 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:26.328056 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:26.328062 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:26.328065 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:26.328169 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:26.328183 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:26.328188 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:26.328252 (+ 64us) tablet.cc:698] Row locks acquired
0110 02:51:26.328255 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:26.328290 (+ 35us) write_op.cc:270] Start()
0110 02:51:26.328325 (+ 35us) write_op.cc:276] Timestamp: P: 1736477486328207 usec, L: 0
0110 02:51:26.328329 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:26.328510 (+ 181us) log.cc:844] Serialized 527 byte log entry
0110 02:51:26.332101 (+ 3591us) op_driver.cc:464] REPLICATION: finished
0110 02:51:26.332240 (+ 139us) write_op.cc:301] APPLY: starting
0110 02:51:26.332254 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:26.332298 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:26.332326 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:26.332376 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:26.332405 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:26.332411 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:26.332481 (+ 70us) log.cc:844] Serialized 23 byte log entry
0110 02:51:26.332537 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.332556 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:26.332562 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:26.332660 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:26.332664 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.048048 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.048147 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.048151 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:27.048154 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:27.048232 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.048242 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.048246 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.048293 (+ 47us) tablet.cc:698] Row locks acquired
0110 02:51:27.048295 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.048320 (+ 25us) write_op.cc:270] Start()
0110 02:51:27.048343 (+ 23us) write_op.cc:276] Timestamp: P: 1736477487048282 usec, L: 0
0110 02:51:27.048346 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.048473 (+ 127us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.055441 (+ 6968us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.055541 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:27.055554 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.055583 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.055610 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.055650 (+ 40us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.055675 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.055680 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:27.055752 (+ 72us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.055799 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.055812 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:27.055817 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.055869 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.055872 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.059441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.059551 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.059559 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:27.059563 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:27.059673 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.059687 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.059695 (+ 8us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.059761 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:27.059764 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.059793 (+ 29us) write_op.cc:270] Start()
0110 02:51:27.059824 (+ 31us) write_op.cc:276] Timestamp: P: 1736477487059752 usec, L: 0
0110 02:51:27.059828 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.060015 (+ 187us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.063091 (+ 3076us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.063189 (+ 98us) write_op.cc:301] APPLY: starting
0110 02:51:27.063201 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.063229 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.063250 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.063287 (+ 37us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.063312 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.063319 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:27.063381 (+ 62us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.063428 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.063441 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:27.063446 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.063508 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.063511 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:27.070714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.070801 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.070806 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.070808 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.070882 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.070891 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.070895 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.070943 (+ 48us) tablet.cc:698] Row locks acquired
0110 02:51:27.070946 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.070971 (+ 25us) write_op.cc:270] Start()
0110 02:51:27.070994 (+ 23us) write_op.cc:276] Timestamp: P: 1736477487070930 usec, L: 0
0110 02:51:27.070997 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.071214 (+ 217us) log.cc:844] Serialized 356 byte log entry
0110 02:51:27.074757 (+ 3543us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.074857 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:27.074867 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.074893 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.074917 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.074954 (+ 37us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.074986 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.074992 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:27.075042 (+ 50us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.075096 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.075109 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:27.075114 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":37026,"reported_tablets":266,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":380,"num_ops":1,"prepare.queue_time_us":293,"prepare.run_cpu_time_us":658,"prepare.run_wall_time_us":659,"raft.queue_time_us":451,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":192,"replication_time_us":3750}],["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":323,"apply.run_wall_time_us":323,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":515,"prepare.run_wall_time_us":515,"raft.queue_time_us":51,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"replication_time_us":7071}],["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":300,"num_ops":1,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":679,"prepare.run_wall_time_us":679,"raft.queue_time_us":74,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":3230}],["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":297,"apply.run_wall_time_us":297,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":527,"prepare.run_wall_time_us":576,"raft.queue_time_us":67,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"replication_time_us":3739}]]}
I20250110 02:51:27.743966 1457 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_09da00340dfa4ef59164793be294d5df" } new_table_name: "default.Table_5f2cdcc0d6c94163b11c6639fdf0b9d5" modify_external_catalogs: true
W20250110 02:51:27.744689 994 leader_election.cc:343] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
I20250110 02:51:27.732040 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:27.748946 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:27.749209 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:27.749365 1188 raft_consensus.cc:738] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.749665 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:27.749672 1188 consensus_queue.cc:260] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.750339 1188 ts_tablet_manager.cc:1428] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250110 02:51:27.750444 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:27.750921 2224 ts_tablet_manager.cc:1918] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.754155 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 171) took 3484 ms. Trace:
I20250110 02:51:27.754287 2352 rpcz_store.cc:276] 0110 02:51:24.269534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269651 (+ 117us) service_pool.cc:225] Handling call
0110 02:51:27.567351 (+3297700us) ts_tablet_manager.cc:1145] Deleting tablet 9507a5283d8f4d07a293ba474c642298
0110 02:51:27.567358 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.671047 (+103689us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.727561 (+ 56514us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.754132 (+ 26571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3297580}
I20250110 02:51:27.688906 1184 raft_consensus.cc:357] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.693027 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 122) took 3442 ms. Trace:
I20250110 02:51:27.705106 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dd1e726f1a41d0894737b3ef3caf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:27.740163 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.755172 2357 rpcz_store.cc:276] 0110 02:51:24.250217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250267 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:27.622914 (+3372647us) ts_tablet_manager.cc:1145] Deleting tablet 6e4d3270e3a04cc99f90943b9573ec7e
0110 02:51:27.622939 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.652502 (+ 29563us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.655212 (+ 2710us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.693008 (+ 37796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3372482}
I20250110 02:51:27.755520 2357 tablet_replica.cc:331] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.755721 2357 raft_consensus.cc:2238] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.755908 2357 raft_consensus.cc:2267] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.756491 2357 ts_tablet_manager.cc:1905] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.755151 1184 raft_consensus.cc:383] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.758855 1184 raft_consensus.cc:738] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.759083 1184 consensus_queue.cc:260] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.754873 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 9507a5283d8f4d07a293ba474c642298 (table default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]) successfully deleted
I20250110 02:51:27.758634 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 6e4d3270e3a04cc99f90943b9573ec7e (table default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]) successfully deleted
I20250110 02:51:27.762789 1184 ts_tablet_manager.cc:1428] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20250110 02:51:27.763432 2357 ts_tablet_manager.cc:1918] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:27.763520 2357 log.cc:1198] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:27.763809 2357 ts_tablet_manager.cc:1939] T d51adff7f50c4aec94e1892a50ec36c4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.764201 1184 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.765619 1184 tablet_bootstrap.cc:654] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.766500 2632 raft_consensus.cc:491] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.766614 2632 raft_consensus.cc:513] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.766849 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.767165 2632 leader_election.cc:290] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.767355 2632 raft_consensus.cc:491] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.767455 2632 raft_consensus.cc:513] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.767925 2632 leader_election.cc:290] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.768414 2632 raft_consensus.cc:491] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:27.768513 2632 raft_consensus.cc:513] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.768946 2632 leader_election.cc:290] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:27.769613 709 leader_election.cc:343] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:27.769693 711 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:27.769709 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:27.769754 709 leader_election.cc:304] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.769943 2632 raft_consensus.cc:2743] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:27.770294 709 leader_election.cc:343] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.770447 709 leader_election.cc:304] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.770658 2632 raft_consensus.cc:2743] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:27.770975 709 leader_election.cc:343] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:27.771241 709 leader_election.cc:343] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.771354 709 leader_election.cc:304] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.771549 2632 raft_consensus.cc:2743] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:27.771890 709 consensus_peers.cc:487] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e25f9643ac0476a90c93fb9aeb921a3. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:27.772109 709 leader_election.cc:343] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.772234 709 leader_election.cc:304] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.772428 2632 raft_consensus.cc:2743] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.772972 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.752606 1181 tablet_bootstrap.cc:654] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.755724 996 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.773404 2224 log.cc:1198] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:27.773685 2224 ts_tablet_manager.cc:1939] T cdb726d2c9a04963b92a3fb625cf84f3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:27.774329 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 143) took 4205 ms. Trace:
I20250110 02:51:27.774436 2224 rpcz_store.cc:276] 0110 02:51:23.569012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.569090 (+ 78us) service_pool.cc:225] Handling call
0110 02:51:27.633723 (+4064633us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:27.633757 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.663815 (+ 30058us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.750799 (+ 86984us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.774311 (+ 23512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4064416}
I20250110 02:51:27.774802 2224 tablet_replica.cc:331] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.774823 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet cdb726d2c9a04963b92a3fb625cf84f3 (table default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]) successfully deleted
I20250110 02:51:27.774971 2224 raft_consensus.cc:2238] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:27.775214 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 107) took 3247 ms (client timeout 1000 ms). Trace:
I20250110 02:51:27.775242 2224 raft_consensus.cc:2267] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.775707 2224 ts_tablet_manager.cc:1905] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.782867 2456 raft_consensus.cc:2743] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:27.784720 2352 tablet_replica.cc:331] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:27.784951 2352 raft_consensus.cc:2238] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.768170 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.768388 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.786939 711 leader_election.cc:343] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.787366 1188 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:27.775303 1439 rpcz_store.cc:269] 0110 02:51:24.527353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.920104 (+392751us) service_pool.cc:225] Handling call
0110 02:51:25.813596 (+893492us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.813601 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:27.775161 (+1961560us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:25.807837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:25.807999 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:25.808006 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:25.808008 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:25.808111 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:25.808124 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:25.808130 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:25.808196 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:25.808199 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:25.808229 (+ 30us) write_op.cc:270] Start()
0110 02:51:25.808278 (+ 49us) write_op.cc:276] Timestamp: P: 1736477485808201 usec, L: 0
0110 02:51:25.808282 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:25.808447 (+ 165us) log.cc:844] Serialized 527 byte log entry
0110 02:51:25.811483 (+ 3036us) op_driver.cc:464] REPLICATION: finished
0110 02:51:25.811576 (+ 93us) write_op.cc:301] APPLY: starting
0110 02:51:25.811589 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:25.811626 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:25.811656 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:25.811700 (+ 44us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:25.811727 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:25.811734 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:25.811793 (+ 59us) log.cc:844] Serialized 23 byte log entry
0110 02:51:25.811858 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:25.811875 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:25.811880 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":308191,"reported_tablets":283,"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":345,"apply.run_wall_time_us":346,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":696,"raft.queue_time_us":488,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":207,"replication_time_us":3178}]]}
W20250110 02:51:27.787947 711 leader_election.cc:343] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.788825 1188 tablet_bootstrap.cc:654] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.789472 2352 raft_consensus.cc:2267] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.789939 2352 ts_tablet_manager.cc:1905] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.791582 1439 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_6156e903bcdd40a4bdcb55ad590629be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:27.792493 1439 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6156e903bcdd40a4bdcb55ad590629be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:27.794018 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 124) took 3543 ms. Trace:
I20250110 02:51:27.794406 2357 rpcz_store.cc:276] 0110 02:51:24.250426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250507 (+ 81us) service_pool.cc:225] Handling call
0110 02:51:27.755473 (+3504966us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:27.755484 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.760301 (+ 4817us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.763363 (+ 3062us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.793999 (+ 30636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3504825}
I20250110 02:51:27.795429 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d51adff7f50c4aec94e1892a50ec36c4 (table default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]) successfully deleted
I20250110 02:51:27.795975 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 123) took 3545 ms. Trace:
I20250110 02:51:27.796434 1181 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.796604 1181 ts_tablet_manager.cc:1397] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.002s
I20250110 02:51:27.796262 2357 rpcz_store.cc:276] 0110 02:51:24.250310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250520 (+ 210us) service_pool.cc:225] Handling call
0110 02:51:27.795435 (+3544915us) ts_tablet_manager.cc:1145] Deleting tablet 4aa78a9ef5eb443da0c8983a7190440b
0110 02:51:27.795445 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.795857 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3544741}
W20250110 02:51:27.797153 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:27.797304 1181 raft_consensus.cc:357] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.797519 1181 raft_consensus.cc:383] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.797601 1181 raft_consensus.cc:738] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.797832 1181 consensus_queue.cc:260] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.797924 2357 tablet_replica.cc:331] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.798528 1181 ts_tablet_manager.cc:1428] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:27.798823 1181 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.799082 2357 raft_consensus.cc:2238] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.799770 2632 raft_consensus.cc:491] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.799919 2632 raft_consensus.cc:513] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.799901 2357 raft_consensus.cc:2267] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.800541 1181 tablet_bootstrap.cc:654] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.800568 2632 leader_election.cc:290] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.801095 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:27.801630 711 leader_election.cc:343] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.803891 2357 ts_tablet_manager.cc:1905] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.805166 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.808380 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.809486 1184 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.809641 1184 ts_tablet_manager.cc:1397] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.001s
I20250110 02:51:27.810293 1184 raft_consensus.cc:357] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.810499 1184 raft_consensus.cc:383] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.810563 1184 raft_consensus.cc:738] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.810739 1184 consensus_queue.cc:260] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.817656 1184 ts_tablet_manager.cc:1428] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250110 02:51:27.817963 1184 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.819002 1439 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 112) took 2547 ms. Trace:
I20250110 02:51:27.819453 1184 tablet_bootstrap.cc:654] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.819159 1439 rpcz_store.cc:276] 0110 02:51:25.271121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.790632 (+2519511us) service_pool.cc:225] Handling call
0110 02:51:27.794049 (+ 3417us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:27.794653 (+ 604us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:27.817957 (+ 23304us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:27.818086 (+ 129us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:27.818221 (+ 135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.803424 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.803529 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.803534 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.803536 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.803602 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.803611 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.803615 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.803672 (+ 57us) tablet.cc:698] Row locks acquired
0110 02:51:27.803674 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.803696 (+ 22us) write_op.cc:270] Start()
0110 02:51:27.803717 (+ 21us) write_op.cc:276] Timestamp: P: 1736477487803642 usec, L: 0
0110 02:51:27.803720 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.803855 (+ 135us) log.cc:844] Serialized 433 byte log entry
0110 02:51:27.806237 (+ 2382us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.806333 (+ 96us) write_op.cc:301] APPLY: starting
0110 02:51:27.806344 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.806401 (+ 57us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.806426 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.806511 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.806540 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:27.806547 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:27.806602 (+ 55us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.806651 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.806665 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:27.806670 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.806731 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.806734 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":386,"apply.run_wall_time_us":391,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":503,"prepare.run_wall_time_us":502,"raft.queue_time_us":47,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"replication_time_us":2494}]]}
W20250110 02:51:27.821650 709 leader_election.cc:343] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.828778 1181 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.828977 1181 ts_tablet_manager.cc:1397] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250110 02:51:27.829787 1181 raft_consensus.cc:357] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.830014 1181 raft_consensus.cc:383] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.830102 1181 raft_consensus.cc:738] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.830345 1181 consensus_queue.cc:260] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.830924 1181 ts_tablet_manager.cc:1428] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:27.831202 1181 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.832669 1181 tablet_bootstrap.cc:654] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.846968 1188 tablet_bootstrap.cc:492] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:27.848129 1188 ts_tablet_manager.cc:1397] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.000s
I20250110 02:51:27.848886 1188 raft_consensus.cc:357] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.849416 709 leader_election.cc:304] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.849730 2549 raft_consensus.cc:2743] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:27.850086 709 leader_election.cc:343] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.850193 709 leader_election.cc:304] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.850364 2549 raft_consensus.cc:2743] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:27.850721 709 leader_election.cc:343] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:27.850852 709 leader_election.cc:304] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.851045 2549 raft_consensus.cc:2743] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 02:51:27.851334 709 leader_election.cc:343] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.851426 709 leader_election.cc:304] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.851591 2549 raft_consensus.cc:2743] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.851743 2549 raft_consensus.cc:491] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.851853 2600 raft_consensus.cc:491] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.851897 2549 raft_consensus.cc:513] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.851989 2600 raft_consensus.cc:513] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.846151 2357 ts_tablet_manager.cc:1918] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.852460 2549 leader_election.cc:290] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.852540 2600 leader_election.cc:290] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.852533 2357 log.cc:1198] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:27.853582 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.853698 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.853883 768 raft_consensus.cc:2463] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:27.853967 711 leader_election.cc:343] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.854238 996 leader_election.cc:304] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.854475 2600 raft_consensus.cc:2798] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.854538 2600 raft_consensus.cc:491] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.854589 2600 raft_consensus.cc:3054] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.854941 2357 ts_tablet_manager.cc:1939] T b2163c343a694f5ca04ac9f7610d25e3 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.853410 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.855926 1188 raft_consensus.cc:383] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.853446 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.856026 1188 raft_consensus.cc:738] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.856216 1188 consensus_queue.cc:260] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.856715 1188 ts_tablet_manager.cc:1428] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250110 02:51:27.856761 2600 raft_consensus.cc:513] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:27.857139 994 leader_election.cc:343] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.858045 2600 leader_election.cc:290] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.858125 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:27.858556 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:27.858623 994 leader_election.cc:343] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.858731 768 raft_consensus.cc:3054] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.861629 768 raft_consensus.cc:2463] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.862185 2549 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85f9597077934f1a9814d284ed06d13d)
I20250110 02:51:27.862355 2549 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.862560 996 leader_election.cc:304] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:27.863106 2549 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.863162 2456 raft_consensus.cc:2798] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.863296 2456 raft_consensus.cc:695] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.863570 2456 consensus_queue.cc:237] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.866523 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.867326 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:27.868129 2352 ts_tablet_manager.cc:1918] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.868649 1056 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:27.869637 711 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.869910 2549 raft_consensus.cc:2798] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 02:51:27.869987 2549 raft_consensus.cc:491] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85f9597077934f1a9814d284ed06d13d)
I20250110 02:51:27.870047 2549 raft_consensus.cc:3054] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 02:51:27.872308 2549 raft_consensus.cc:513] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.872745 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:27.872825 2549 leader_election.cc:290] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.873104 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:27.873346 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7885a80414f2193400672aa16a911" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:27.873454 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:27.873566 1056 raft_consensus.cc:3054] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Advancing to term 2
I20250110 02:51:27.873951 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:27.874177 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:27.874496 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:27.876084 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:27.876302 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:27.876525 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:27.876713 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:27.876925 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:27.877391 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:27.878051 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:27.878571 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
W20250110 02:51:27.879483 709 leader_election.cc:343] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.879624 709 leader_election.cc:304] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.879848 2549 raft_consensus.cc:2743] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.881726 1056 raft_consensus.cc:2463] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 2.
I20250110 02:51:27.883402 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 126) took 3632 ms. Trace:
I20250110 02:51:27.883497 2357 rpcz_store.cc:276] 0110 02:51:24.250645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250732 (+ 87us) service_pool.cc:225] Handling call
0110 02:51:27.797794 (+3547062us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:27.797893 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.823483 (+ 25590us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.845538 (+ 22055us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.883383 (+ 37845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3546922}
I20250110 02:51:27.883783 2357 tablet_replica.cc:331] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:27.883414 709 leader_election.cc:343] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
W20250110 02:51:27.884198 709 leader_election.cc:343] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
I20250110 02:51:27.883950 2357 raft_consensus.cc:2238] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.884406 2357 raft_consensus.cc:2267] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.884914 2357 ts_tablet_manager.cc:1905] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.886329 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b2163c343a694f5ca04ac9f7610d25e3 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:27.886584 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:27.886741 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:27.886905 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:27.887404 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet abbd43638a2c47dcba4f6f90e8676f42 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]). Expected version 1 got 0
I20250110 02:51:27.887380 711 leader_election.cc:304] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:27.887575 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:27.887778 2549 raft_consensus.cc:2798] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 02:51:27.887893 2549 raft_consensus.cc:695] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 2 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:27.887933 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:27.888132 2549 consensus_queue.cc:237] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.888615 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:27.888877 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:27.889096 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:27.889492 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:27.890707 1184 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:27.890839 1184 ts_tablet_manager.cc:1397] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250110 02:51:27.891582 1181 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.891505 1184 raft_consensus.cc:357] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.891726 1181 ts_tablet_manager.cc:1397] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.005s
I20250110 02:51:27.892243 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:27.892354 1184 raft_consensus.cc:383] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.892530 1184 raft_consensus.cc:738] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.892540 1181 raft_consensus.cc:357] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.892727 1181 raft_consensus.cc:383] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.892784 1181 raft_consensus.cc:738] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.892762 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:27.892793 1184 consensus_queue.cc:260] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:27.893278 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:27.893532 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:27.893007 1181 consensus_queue.cc:260] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.894182 1184 ts_tablet_manager.cc:1428] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:27.894268 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:27.894546 1181 ts_tablet_manager.cc:1428] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:27.894795 1181 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.896587 1181 tablet_bootstrap.cc:654] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.898311 2600 raft_consensus.cc:491] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.898434 2600 raft_consensus.cc:513] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.899044 2600 leader_election.cc:290] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.900431 2600 raft_consensus.cc:491] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.900523 2600 raft_consensus.cc:513] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.900960 2600 leader_election.cc:290] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.871752 1188 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:27.901839 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.902014 768 raft_consensus.cc:2463] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.902253 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.902438 764 raft_consensus.cc:2463] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:27.902419 996 leader_election.cc:304] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.902699 2456 raft_consensus.cc:2798] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.902765 2456 raft_consensus.cc:491] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.902827 2456 raft_consensus.cc:3054] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.902860 1188 tablet_bootstrap.cc:654] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.902941 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:27.899672 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.901485 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:27.903605 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:27.904517 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:27.905550 2456 raft_consensus.cc:513] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.906836 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:27.906880 994 leader_election.cc:343] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:27.907182 994 leader_election.cc:343] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.907801 2456 leader_election.cc:290] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.907919 1184 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:27.908063 996 leader_election.cc:304] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:27.908471 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.908895 764 raft_consensus.cc:3054] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.908924 2289 raft_consensus.cc:2798] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:27.909001 2289 raft_consensus.cc:491] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:27.909032 2352 log.cc:1198] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:27.909061 2289 raft_consensus.cc:3054] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.909405 2352 ts_tablet_manager.cc:1939] T e744cdc1fcb94d46b48b684763ecf1bf P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:27.909655 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 172) took 3640 ms. Trace:
I20250110 02:51:27.909711 2352 rpcz_store.cc:276] 0110 02:51:24.269661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269796 (+ 135us) service_pool.cc:225] Handling call
0110 02:51:27.784669 (+3514873us) ts_tablet_manager.cc:1145] Deleting tablet e744cdc1fcb94d46b48b684763ecf1bf
0110 02:51:27.784680 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.864894 (+ 80214us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.868030 (+ 3136us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.909644 (+ 41614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3514762}
I20250110 02:51:27.911372 764 raft_consensus.cc:2463] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.911361 2289 raft_consensus.cc:513] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.911777 2289 leader_election.cc:290] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.910116 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e744cdc1fcb94d46b48b684763ecf1bf (table default.Table_83ab4512090d4fd0b699700953b61a82 [id=16b3b55134c441d881d183c9a90a35f3]) successfully deleted
I20250110 02:51:27.912353 996 leader_election.cc:304] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:27.912436 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:27.912514 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:27.912642 764 raft_consensus.cc:3054] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:27.912662 994 leader_election.cc:343] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.912757 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:27.913313 2600 raft_consensus.cc:2798] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.913407 2600 raft_consensus.cc:695] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.913585 2600 consensus_queue.cc:237] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.914502 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:27.914945 994 leader_election.cc:343] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.917382 764 raft_consensus.cc:2463] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:27.917986 996 leader_election.cc:304] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:27.918197 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:27.918785 1440 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:27.919880 2456 raft_consensus.cc:2798] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:27.920068 2456 raft_consensus.cc:695] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:27.920363 2456 consensus_queue.cc:237] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.922627 2352 tablet_replica.cc:331] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:27.926627 2352 raft_consensus.cc:2238] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.927342 2352 raft_consensus.cc:2267] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.927721 2224 ts_tablet_manager.cc:1918] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:27.933132 2352 ts_tablet_manager.cc:1905] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.936419 2224 log.cc:1198] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:27.936842 2224 ts_tablet_manager.cc:1939] T 65ae1c19d2dc4778a1cebddbd505f907 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:27.942559 853 consensus_peers.cc:487] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:27.947150 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 147) took 4375 ms. Trace:
I20250110 02:51:27.936229 1181 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.948372 2224 rpcz_store.cc:276] 0110 02:51:23.571829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.572579 (+ 750us) service_pool.cc:225] Handling call
0110 02:51:27.774765 (+4202186us) ts_tablet_manager.cc:1145] Deleting tablet 65ae1c19d2dc4778a1cebddbd505f907
0110 02:51:27.774773 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.875450 (+100677us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.927627 (+ 52177us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.947133 (+ 19506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4202009}
I20250110 02:51:27.948587 1181 ts_tablet_manager.cc:1397] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.000s
I20250110 02:51:27.948834 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 65ae1c19d2dc4778a1cebddbd505f907 (table default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]) successfully deleted
I20250110 02:51:27.949260 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 144) took 4377 ms. Trace:
I20250110 02:51:27.949358 1181 raft_consensus.cc:357] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.949581 1181 raft_consensus.cc:383] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.949669 1181 raft_consensus.cc:738] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.949925 1181 consensus_queue.cc:260] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.950062 1188 tablet_bootstrap.cc:492] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:27.950183 1188 ts_tablet_manager.cc:1397] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.004s
I20250110 02:51:27.950562 1181 ts_tablet_manager.cc:1428] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:27.950847 1181 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.950798 1188 raft_consensus.cc:357] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.951004 1188 raft_consensus.cc:383] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.951086 1188 raft_consensus.cc:738] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.951303 1188 consensus_queue.cc:260] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.951882 1188 ts_tablet_manager.cc:1428] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:27.952265 1188 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:27.952452 1181 tablet_bootstrap.cc:654] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.956287 881 tablet_service.cc:1467] Processing CreateTablet for tablet c13eb353c61c48e08328ee9e7fb3b32a (DEFAULT_TABLE table=default.Table_6156e903bcdd40a4bdcb55ad590629be [id=3f95c61eec4a468d9a4d208b87857fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:27.956604 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13eb353c61c48e08328ee9e7fb3b32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:27.959750 1188 tablet_bootstrap.cc:654] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.961894 1030 tablet_service.cc:1467] Processing CreateTablet for tablet c13eb353c61c48e08328ee9e7fb3b32a (DEFAULT_TABLE table=default.Table_6156e903bcdd40a4bdcb55ad590629be [id=3f95c61eec4a468d9a4d208b87857fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:27.962220 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13eb353c61c48e08328ee9e7fb3b32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:27.962146 754 tablet_service.cc:1467] Processing CreateTablet for tablet c13eb353c61c48e08328ee9e7fb3b32a (DEFAULT_TABLE table=default.Table_6156e903bcdd40a4bdcb55ad590629be [id=3f95c61eec4a468d9a4d208b87857fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:27.962466 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13eb353c61c48e08328ee9e7fb3b32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:27.949371 2224 rpcz_store.cc:276] 0110 02:51:23.571486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.572912 (+ 1426us) service_pool.cc:225] Handling call
0110 02:51:27.949136 (+4376224us) ts_tablet_manager.cc:1145] Deleting tablet a0ddab9e847445999ec97eac4352d9c2
0110 02:51:27.949145 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.949232 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4375942}
W20250110 02:51:27.964982 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:27.966743 2224 tablet_replica.cc:331] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:27.966964 2224 raft_consensus.cc:2238] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:27.967288 2224 raft_consensus.cc:2267] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.968467 2357 ts_tablet_manager.cc:1918] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:27.968461 2504 log.cc:926] Time spent T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Append to log took a long time: real 0.056s user 0.002s sys 0.000s
I20250110 02:51:27.968600 2357 log.cc:1198] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:27.968942 2357 ts_tablet_manager.cc:1939] T e3fc9826ef554a5699e365d10d02e97a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:27.969434 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 110) took 3720 ms. Trace:
I20250110 02:51:27.969543 2357 rpcz_store.cc:276] 0110 02:51:24.248900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250789 (+ 1889us) service_pool.cc:225] Handling call
0110 02:51:27.883743 (+3632954us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:27.883752 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.933772 (+ 50020us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.968391 (+ 34619us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:27.969418 (+ 1027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3632851}
I20250110 02:51:27.970000 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e3fc9826ef554a5699e365d10d02e97a (table default.Table_64d72c69126e4192abb401ad9e536c0a [id=22e270395ace4a27bd3ce85c2e17c9ff]) successfully deleted
I20250110 02:51:27.970324 2357 tablet_replica.cc:331] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:27.970522 2357 raft_consensus.cc:2238] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:27.970650 2357 raft_consensus.cc:2267] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:27.971100 2377 raft_consensus.cc:491] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.971208 2377 raft_consensus.cc:513] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.971726 2377 leader_election.cc:290] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:27.971879 2549 raft_consensus.cc:491] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:27.971984 2549 raft_consensus.cc:513] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.972579 2549 leader_election.cc:290] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:27.972828 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.973055 1056 raft_consensus.cc:2388] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:27.973281 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:27.973874 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:27.973966 711 leader_election.cc:343] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:27.974514 709 leader_election.cc:343] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:27.974601 709 leader_election.cc:304] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:27.974987 2549 raft_consensus.cc:2743] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.975689 2224 ts_tablet_manager.cc:1905] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.979166 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ca5369caf3415ea508352362409656" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:27.979889 853 leader_election.cc:304] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:27.980160 2377 raft_consensus.cc:3054] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:27.980482 2456 consensus_queue.cc:1035] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.982227 2377 raft_consensus.cc:2743] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:27.985479 1184 tablet_bootstrap.cc:654] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:27.987316 2478 consensus_queue.cc:1035] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:27.992067 2357 ts_tablet_manager.cc:1905] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:27.997943 1181 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:27.998099 1181 ts_tablet_manager.cc:1397] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20250110 02:51:27.999056 1181 raft_consensus.cc:357] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:27.999349 1181 raft_consensus.cc:383] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:27.999426 1181 raft_consensus.cc:738] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:27.999645 1181 consensus_queue.cc:260] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.000133 1181 ts_tablet_manager.cc:1428] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:28.000356 1181 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:27.992864 1184 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.001504 1184 ts_tablet_manager.cc:1397] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.001s
I20250110 02:51:28.002111 1184 raft_consensus.cc:357] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.002277 1184 raft_consensus.cc:383] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.002349 1184 raft_consensus.cc:738] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.002586 1184 consensus_queue.cc:260] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.003171 1184 ts_tablet_manager.cc:1428] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:28.003414 1184 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.004770 1184 tablet_bootstrap.cc:654] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.005522 2224 ts_tablet_manager.cc:1918] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.005630 2224 log.cc:1198] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1fb5412b78104d8085878c58b191ab27
I20250110 02:51:28.005990 2224 ts_tablet_manager.cc:1939] T 1fb5412b78104d8085878c58b191ab27 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.006525 2352 ts_tablet_manager.cc:1918] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.006639 2352 log.cc:1198] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:28.006939 2352 ts_tablet_manager.cc:1939] T 747ae8cc8c2c4ffa9071e23f7d7c5e11 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.007228 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 167) took 3738 ms. Trace:
I20250110 02:51:28.007310 2352 rpcz_store.cc:276] 0110 02:51:24.269126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269930 (+ 804us) service_pool.cc:225] Handling call
0110 02:51:27.922582 (+3652652us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:27.922592 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:27.944636 (+ 22044us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.006437 (+ 61801us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.007205 (+ 768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3652537}
I20250110 02:51:28.007725 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]) successfully deleted
I20250110 02:51:28.010782 1188 tablet_bootstrap.cc:492] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.011019 1188 ts_tablet_manager.cc:1397] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20250110 02:51:28.011727 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:28.011754 1188 raft_consensus.cc:357] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.012058 1188 raft_consensus.cc:383] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.012180 1188 raft_consensus.cc:738] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.012477 1188 consensus_queue.cc:260] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.013180 1188 ts_tablet_manager.cc:1428] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:28.013571 1188 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.015885 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 158) took 3747 ms. Trace:
I20250110 02:51:28.015995 2352 rpcz_store.cc:276] 0110 02:51:24.268210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.269937 (+ 1727us) service_pool.cc:225] Handling call
0110 02:51:28.015783 (+3745846us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:28.015791 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.015871 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3745668}
I20250110 02:51:28.016402 2352 tablet_replica.cc:331] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:28.016427 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80186dd7d1a240068584b64ce9e9b03f already in progress: creating tablet
I20250110 02:51:28.016577 2352 raft_consensus.cc:2238] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.016773 2352 raft_consensus.cc:2267] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.016799 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:28.017172 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:28.017237 2352 ts_tablet_manager.cc:1905] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.017385 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:28.017576 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:28.017798 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:28.018018 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:28.018291 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:28.018719 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:28.024875 2357 ts_tablet_manager.cc:1918] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:28.027563 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 148) took 4454 ms. Trace:
I20250110 02:51:28.027778 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:28.027899 2224 rpcz_store.cc:276] 0110 02:51:23.573052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.573269 (+ 217us) service_pool.cc:225] Handling call
0110 02:51:27.966640 (+4393371us) ts_tablet_manager.cc:1145] Deleting tablet 1fb5412b78104d8085878c58b191ab27
0110 02:51:27.966649 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.002744 (+ 36095us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.005443 (+ 2699us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.027542 (+ 22099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4393250}
I20250110 02:51:28.028053 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:28.028266 2224 tablet_replica.cc:331] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.028475 2224 raft_consensus.cc:2238] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.028667 2224 raft_consensus.cc:2267] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.029145 2224 ts_tablet_manager.cc:1905] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.029289 2357 log.cc:1198] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:28.029595 2357 ts_tablet_manager.cc:1939] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.030511 1188 tablet_bootstrap.cc:654] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.030064 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1fb5412b78104d8085878c58b191ab27 (table default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]) successfully deleted
I20250110 02:51:28.034643 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:28.034997 2352 ts_tablet_manager.cc:1918] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.034935 1439 catalog_manager.cc:5526] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f9597077934f1a9814d284ed06d13d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "85f9597077934f1a9814d284ed06d13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:28.035439 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 9b937188bf2949af8f54bade4e0ad984 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:28.035764 2352 log.cc:1198] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/585320c0970c4097b195caaae33789b5
I20250110 02:51:28.035959 1184 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.035952 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:28.036072 1184 ts_tablet_manager.cc:1397] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250110 02:51:28.036244 2352 ts_tablet_manager.cc:1939] T 585320c0970c4097b195caaae33789b5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.036564 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:28.036687 1184 raft_consensus.cc:357] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.036901 1184 raft_consensus.cc:383] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.036986 1184 raft_consensus.cc:738] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.037173 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:28.037245 1184 consensus_queue.cc:260] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.038159 1184 ts_tablet_manager.cc:1428] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:28.038177 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:28.038491 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_72d32c146dce4aac925f27638bf13312 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 2 got 0
I20250110 02:51:28.038949 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_56ef844e399040a3ac6279fc5d2ed20b [id=78b013e77cf340089467193ef7c17819]). Expected version 1 got 0
I20250110 02:51:28.039161 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:28.039626 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:28.039703 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 169) took 3770 ms. Trace:
I20250110 02:51:28.040197 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:28.040618 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:28.040823 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:28.041381 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:28.042225 1184 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.043618 1184 tablet_bootstrap.cc:654] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.045742 1181 tablet_bootstrap.cc:654] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.047111 2478 raft_consensus.cc:491] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.047235 2478 raft_consensus.cc:513] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.047407 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:28.047896 2478 leader_election.cc:290] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.048153 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:28.048396 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:28.048470 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.048532 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:28.048638 764 raft_consensus.cc:2463] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.049068 996 leader_election.cc:304] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:28.049264 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:28.049413 2600 raft_consensus.cc:2798] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.049418 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:28.049485 2600 raft_consensus.cc:491] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.049554 2600 raft_consensus.cc:3054] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.049657 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:28.050186 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:28.051141 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:28.051568 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:28.051937 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:28.052281 2600 raft_consensus.cc:513] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.053035 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.053187 764 raft_consensus.cc:3054] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.055605 764 raft_consensus.cc:2463] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.056867 996 leader_election.cc:304] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:28.057622 2024 raft_consensus.cc:2798] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.057736 2024 raft_consensus.cc:695] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.057917 2024 consensus_queue.cc:237] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.060031 2600 leader_election.cc:290] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.061934 1457 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 111) took 2809 ms. Trace:
I20250110 02:51:28.062045 1457 rpcz_store.cc:276] 0110 02:51:25.251994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.743744 (+2491750us) service_pool.cc:225] Handling call
0110 02:51:27.744181 (+ 437us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.744388 (+ 207us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:27.744586 (+ 198us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:27.744665 (+ 79us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:27.750457 (+ 5792us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:28.061899 (+311442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.745161 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.745252 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.745257 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.745259 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.745338 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.745348 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.745352 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:27.745397 (+ 45us) tablet.cc:698] Row locks acquired
0110 02:51:27.745399 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.745427 (+ 28us) write_op.cc:270] Start()
0110 02:51:27.745452 (+ 25us) write_op.cc:276] Timestamp: P: 1736477487745377 usec, L: 0
0110 02:51:27.745455 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.745613 (+ 158us) log.cc:844] Serialized 405 byte log entry
0110 02:51:27.748294 (+ 2681us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.748403 (+ 109us) write_op.cc:301] APPLY: starting
0110 02:51:27.748417 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.748459 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.748504 (+ 45us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.748550 (+ 46us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.748578 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:27.748586 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:27.748675 (+ 89us) log.cc:844] Serialized 23 byte log entry
0110 02:51:27.748728 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.748744 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:27.748750 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.748828 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.748832 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":418,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":557,"raft.queue_time_us":369,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":196,"replication_time_us":2813}]]}
I20250110 02:51:28.062752 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 111) took 3813 ms. Trace:
I20250110 02:51:28.062822 2357 rpcz_store.cc:276] 0110 02:51:24.249016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250915 (+ 1899us) service_pool.cc:225] Handling call
0110 02:51:27.970288 (+3719373us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:27.970296 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.015571 (+ 45275us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.024801 (+ 9230us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.062737 (+ 37936us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719266}
I20250110 02:51:28.062810 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7c6062aaaae94525b7b305bd8fccc78f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.063367 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c6062aaaae94525b7b305bd8fccc78f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.063855 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 127) took 3813 ms. Trace:
I20250110 02:51:28.063935 2357 rpcz_store.cc:276] 0110 02:51:24.250743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.250931 (+ 188us) service_pool.cc:225] Handling call
0110 02:51:28.063765 (+3812834us) ts_tablet_manager.cc:1145] Deleting tablet 35ec1dba54a247d5ba9f9e053aa3b645
0110 02:51:28.063775 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.063840 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812674}
I20250110 02:51:28.064630 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 19b90cc15dbf47d4b4dbaf12d588abb0 (table default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]) successfully deleted
I20250110 02:51:28.064713 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 114) took 3815 ms. Trace:
I20250110 02:51:28.064766 2357 rpcz_store.cc:276] 0110 02:51:24.249319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.251046 (+ 1727us) service_pool.cc:225] Handling call
0110 02:51:28.064625 (+3813579us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:28.064633 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.064700 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3813475}
W20250110 02:51:28.064844 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 35ec1dba54a247d5ba9f9e053aa3b645 because the tablet was not found. No further retry: Not found: Tablet not found: 35ec1dba54a247d5ba9f9e053aa3b645
W20250110 02:51:28.065029 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
I20250110 02:51:28.065438 2357 tablet_replica.cc:331] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.065658 2357 raft_consensus.cc:2238] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.065815 2357 raft_consensus.cc:2267] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.066259 2357 ts_tablet_manager.cc:1905] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.071740 1184 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.072114 1184 ts_tablet_manager.cc:1397] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20250110 02:51:28.048951 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.058681 2377 raft_consensus.cc:491] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.072499 2377 raft_consensus.cc:513] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.072955 1184 raft_consensus.cc:357] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.073048 2377 leader_election.cc:290] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.073112 1184 raft_consensus.cc:383] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.059234 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:28.073158 1184 raft_consensus.cc:738] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.073467 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.073643 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0b39e2a5a4467ba44926c26d2fd1b2" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:28.074304 994 leader_election.cc:343] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:28.074682 994 leader_election.cc:343] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.074689 853 leader_election.cc:304] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.074785 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 113) took 2770 ms. Trace:
I20250110 02:51:28.074796 1184 consensus_queue.cc:260] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.074899 2352 rpcz_store.cc:276] 0110 02:51:24.269322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270143 (+ 821us) service_pool.cc:225] Handling call
0110 02:51:28.016367 (+3746224us) ts_tablet_manager.cc:1145] Deleting tablet 585320c0970c4097b195caaae33789b5
0110 02:51:28.016374 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.020138 (+ 3764us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.034926 (+ 14788us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.039688 (+ 4762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3746103}
I20250110 02:51:28.075058 2377 raft_consensus.cc:2743] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.061962 1188 tablet_bootstrap.cc:492] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.075309 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 585320c0970c4097b195caaae33789b5 (table default.Table_eea9dbdd7e284980bec035a0b1dc0c4f [id=dae78aa8aadd4ceb81461d6702d32ad9]) successfully deleted
I20250110 02:51:28.075378 1188 ts_tablet_manager.cc:1397] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.001s
I20250110 02:51:28.074892 1457 rpcz_store.cc:276] 0110 02:51:25.304334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.062607 (+2758273us) service_pool.cc:225] Handling call
0110 02:51:28.063458 (+ 851us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.063881 (+ 423us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.074563 (+ 10682us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.074683 (+ 120us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.074754 (+ 71us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.067782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.067878 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.067883 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.067885 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.067956 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.067966 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.067970 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.068030 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:28.068038 (+ 8us) write_op.cc:260] PREPARE: finished
0110 02:51:28.068065 (+ 27us) write_op.cc:270] Start()
0110 02:51:28.068111 (+ 46us) write_op.cc:276] Timestamp: P: 1736477488068028 usec, L: 0
0110 02:51:28.068115 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.068304 (+ 189us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.072178 (+ 3874us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.072277 (+ 99us) write_op.cc:301] APPLY: starting
0110 02:51:28.072289 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.072342 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.072370 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.072462 (+ 92us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.072493 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.072501 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:28.072565 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.072627 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.072643 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:28.072648 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.072707 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.072710 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":423,"num_ops":2,"prepare.queue_time_us":3458,"prepare.run_cpu_time_us":611,"prepare.run_wall_time_us":610,"raft.queue_time_us":824,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":221,"replication_time_us":4038}]]}
I20250110 02:51:28.075627 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 155) took 3807 ms. Trace:
I20250110 02:51:28.075935 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_4cffba7d34a0456f9f600de97d6a1737"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.076153 1184 ts_tablet_manager.cc:1428] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250110 02:51:28.076267 1188 raft_consensus.cc:357] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.076782 1188 raft_consensus.cc:383] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 02:51:28.076927 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cffba7d34a0456f9f600de97d6a1737 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.076915 1188 raft_consensus.cc:738] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.077219 1188 consensus_queue.cc:260] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.077653 1184 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.077968 1188 ts_tablet_manager.cc:1428] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:28.078323 1188 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.081219 1188 tablet_bootstrap.cc:654] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.083696 1184 tablet_bootstrap.cc:654] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.083891 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 114) took 2737 ms. Trace:
I20250110 02:51:28.076642 2352 rpcz_store.cc:276] 0110 02:51:24.267955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270212 (+ 2257us) service_pool.cc:225] Handling call
0110 02:51:28.075542 (+3805330us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:28.075550 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.075615 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3805244}
W20250110 02:51:28.084585 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00e97e496b54dcf82b7182591d6f4aa already in progress: creating tablet
I20250110 02:51:28.084026 1457 rpcz_store.cc:276] 0110 02:51:25.346635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.075749 (+2729114us) service_pool.cc:225] Handling call
0110 02:51:28.077042 (+ 1293us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.077508 (+ 466us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.083632 (+ 6124us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.083639 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:28.083691 (+ 52us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.083781 (+ 90us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.083856 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.077951 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.078057 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.078063 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.078066 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.078161 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.078172 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.078176 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.078238 (+ 62us) tablet.cc:698] Row locks acquired
0110 02:51:28.078240 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.078268 (+ 28us) write_op.cc:270] Start()
0110 02:51:28.078292 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488078228 usec, L: 0
0110 02:51:28.078295 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.078455 (+ 160us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.081172 (+ 2717us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.081279 (+ 107us) write_op.cc:301] APPLY: starting
0110 02:51:28.081291 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.081322 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.081342 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.081431 (+ 89us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.081458 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.081464 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.081522 (+ 58us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.081570 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.081588 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:28.081594 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":577,"prepare.run_wall_time_us":577,"raft.queue_time_us":79,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":189,"replication_time_us":2854}]]}
I20250110 02:51:28.085456 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_67fee57b82044957be4b163808d12438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.086016 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67fee57b82044957be4b163808d12438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.086002 2224 ts_tablet_manager.cc:1918] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:28.086117 2224 log.cc:1198] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:28.087229 2352 tablet_replica.cc:331] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.089491 2352 raft_consensus.cc:2238] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.090505 2352 raft_consensus.cc:2267] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.093312 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 115) took 2637 ms. Trace:
I20250110 02:51:28.093472 1457 rpcz_store.cc:276] 0110 02:51:25.456012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.085251 (+2629239us) service_pool.cc:225] Handling call
0110 02:51:28.086133 (+ 882us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.086587 (+ 454us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.093049 (+ 6462us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.093054 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:28.093108 (+ 54us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.093205 (+ 97us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.093277 (+ 72us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.087151 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.087240 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.087262 (+ 22us) write_op.cc:435] Acquired schema lock
0110 02:51:28.087264 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.087353 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.087365 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.087369 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.087438 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:28.087442 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.087475 (+ 33us) write_op.cc:270] Start()
0110 02:51:28.087502 (+ 27us) write_op.cc:276] Timestamp: P: 1736477488087434 usec, L: 0
0110 02:51:28.087505 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.087673 (+ 168us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.090627 (+ 2954us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.090740 (+ 113us) write_op.cc:301] APPLY: starting
0110 02:51:28.090755 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.090789 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.090820 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.090929 (+ 109us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.090957 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.090962 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:28.091025 (+ 63us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.091073 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.091089 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:28.091094 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":399,"num_ops":2,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":595,"prepare.run_wall_time_us":595,"raft.queue_time_us":71,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":189,"replication_time_us":3097}]]}
I20250110 02:51:28.093744 2352 ts_tablet_manager.cc:1905] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.094807 2224 ts_tablet_manager.cc:1939] T 04e05a7554a14805bdcbd1824a5bd9d5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.094730 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_5ced3e3687f44570880e92ecec188c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.095902 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 149) took 4522 ms. Trace:
I20250110 02:51:28.101373 2224 rpcz_store.cc:276] 0110 02:51:23.573144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.573447 (+ 303us) service_pool.cc:225] Handling call
0110 02:51:28.028218 (+4454771us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:28.028228 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.034287 (+ 6059us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.085891 (+ 51604us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.095880 (+ 9989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4454671}
I20250110 02:51:28.104135 2357 ts_tablet_manager.cc:1918] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.104234 2357 log.cc:1198] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8f4778347f054b86af697dbe3b287263
I20250110 02:51:28.104555 2357 ts_tablet_manager.cc:1939] T 8f4778347f054b86af697dbe3b287263 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.109902 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 118) took 3859 ms. Trace:
I20250110 02:51:28.110800 2357 rpcz_store.cc:276] 0110 02:51:24.249761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.251179 (+ 1418us) service_pool.cc:225] Handling call
0110 02:51:28.065393 (+3814214us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:28.065403 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.081691 (+ 16288us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.104000 (+ 22309us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.109585 (+ 5585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3814109}
I20250110 02:51:28.119168 2357 tablet_replica.cc:331] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.121870 2357 raft_consensus.cc:2238] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.123216 2357 raft_consensus.cc:2267] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.095141 1443 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet dc6b2654f86e4686b8a3034de02d11bf
I20250110 02:51:28.125167 1443 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 106) took 2999 ms. Trace:
I20250110 02:51:28.125267 1443 rpcz_store.cc:276] 0110 02:51:25.126120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.622756 (+2496636us) service_pool.cc:225] Handling call
0110 02:51:27.623269 (+ 513us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.623504 (+ 235us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:27.623580 (+ 76us) catalog_manager.cc:2762] Locking tablets
0110 02:51:27.623740 (+ 160us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:27.661411 (+ 37671us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:27.661490 (+ 79us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:28.095095 (+433605us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:28.125107 (+ 30012us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.624914 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.625072 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.625091 (+ 19us) write_op.cc:435] Acquired schema lock
0110 02:51:27.625095 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:27.625237 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.625252 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.625258 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.625397 (+ 139us) tablet.cc:698] Row locks acquired
0110 02:51:27.625401 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:27.625438 (+ 37us) write_op.cc:270] Start()
0110 02:51:27.625476 (+ 38us) write_op.cc:276] Timestamp: P: 1736477487625397 usec, L: 0
0110 02:51:27.625481 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.625682 (+ 201us) log.cc:844] Serialized 731 byte log entry
0110 02:51:27.660883 (+ 35201us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.661009 (+ 126us) write_op.cc:301] APPLY: starting
0110 02:51:27.661022 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.661065 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.661096 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.661151 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.661184 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:27.661190 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:27.661259 (+ 69us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.661315 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.661331 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:27.661336 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.661407 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.661410 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":391,"num_ops":2,"prepare.queue_time_us":367,"prepare.run_cpu_time_us":871,"prepare.run_wall_time_us":872,"raft.queue_time_us":437,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":259,"replication_time_us":35371}]]}
I20250110 02:51:28.126130 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_6e05f69028df49f0afb60b28409b917a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.095620 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 33f9b3e74ee2431a85a22fefaacfa40e
I20250110 02:51:28.126999 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 108) took 2937 ms. Trace:
I20250110 02:51:28.127090 1448 rpcz_store.cc:276] 0110 02:51:25.189159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.707126 (+2517967us) service_pool.cc:225] Handling call
0110 02:51:27.707709 (+ 583us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.707863 (+ 154us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:27.707942 (+ 79us) catalog_manager.cc:2762] Locking tablets
0110 02:51:27.772596 (+ 64654us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:27.777766 (+ 5170us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:27.777857 (+ 91us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:28.095563 (+317706us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:28.126970 (+ 31407us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.773248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.773334 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.773339 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.773341 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:27.773440 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.773452 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.773456 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.773545 (+ 89us) tablet.cc:698] Row locks acquired
0110 02:51:27.773547 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.773576 (+ 29us) write_op.cc:270] Start()
0110 02:51:27.773600 (+ 24us) write_op.cc:276] Timestamp: P: 1736477487773523 usec, L: 0
0110 02:51:27.773604 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.773745 (+ 141us) log.cc:844] Serialized 725 byte log entry
0110 02:51:27.776734 (+ 2989us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.776856 (+ 122us) write_op.cc:301] APPLY: starting
0110 02:51:27.776869 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.776905 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.776932 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.776988 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.777014 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:27.777019 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:27.777077 (+ 58us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.777140 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.777156 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:27.777161 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.777219 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.777222 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":354,"num_ops":2,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":575,"prepare.run_wall_time_us":575,"raft.queue_time_us":340,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":194,"replication_time_us":3108}]]}
I20250110 02:51:28.127444 2357 ts_tablet_manager.cc:1905] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.127776 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_51799375e04649b5955644f75e31c140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.128286 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51799375e04649b5955644f75e31c140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.095768 1449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet f69d0c201c524b7e95c9d3851b8dd8c2
I20250110 02:51:28.129673 1449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 110) took 2877 ms. Trace:
I20250110 02:51:28.129748 1449 rpcz_store.cc:276] 0110 02:51:25.251904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.735347 (+2483443us) service_pool.cc:225] Handling call
0110 02:51:27.736483 (+ 1136us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.736619 (+ 136us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:27.736688 (+ 69us) catalog_manager.cc:2762] Locking tablets
0110 02:51:27.795908 (+ 59220us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:27.800843 (+ 4935us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:27.800914 (+ 71us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:28.095737 (+294823us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:28.129626 (+ 33889us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.796580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.796668 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.796673 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.796674 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:27.796772 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.796784 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.796791 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.796885 (+ 94us) tablet.cc:698] Row locks acquired
0110 02:51:27.796887 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.796917 (+ 30us) write_op.cc:270] Start()
0110 02:51:27.796943 (+ 26us) write_op.cc:276] Timestamp: P: 1736477487796862 usec, L: 0
0110 02:51:27.796947 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.797121 (+ 174us) log.cc:844] Serialized 759 byte log entry
0110 02:51:27.800195 (+ 3074us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.800282 (+ 87us) write_op.cc:301] APPLY: starting
0110 02:51:27.800295 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.800329 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.800360 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.800409 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.800437 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:27.800442 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:27.800501 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.800549 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.800563 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:27.800568 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.800624 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.800627 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":335,"apply.run_wall_time_us":335,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":627,"prepare.run_wall_time_us":627,"raft.queue_time_us":53,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"replication_time_us":3223}]]}
I20250110 02:51:28.130466 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_5a788687cf5a4bcf9e6f19e3622f3a56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.130859 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a788687cf5a4bcf9e6f19e3622f3a56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.095937 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet d6acae633eb14e5db87b1e36a069422c
I20250110 02:51:28.132019 1456 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 107) took 2967 ms. Trace:
I20250110 02:51:28.132114 1456 rpcz_store.cc:276] 0110 02:51:25.165024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.639337 (+2474313us) service_pool.cc:225] Handling call
0110 02:51:27.639990 (+ 653us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.640189 (+ 199us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:27.640268 (+ 79us) catalog_manager.cc:2762] Locking tablets
0110 02:51:27.640464 (+ 196us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:27.673982 (+ 33518us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:27.675942 (+ 1960us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:28.095889 (+419947us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:28.131994 (+ 36105us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.641565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.641686 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.641694 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:27.641698 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:27.641828 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.641839 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.641846 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.641975 (+ 129us) tablet.cc:698] Row locks acquired
0110 02:51:27.641979 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:27.642008 (+ 29us) write_op.cc:270] Start()
0110 02:51:27.642034 (+ 26us) write_op.cc:276] Timestamp: P: 1736477487641968 usec, L: 0
0110 02:51:27.642037 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.642189 (+ 152us) log.cc:844] Serialized 725 byte log entry
0110 02:51:27.665887 (+ 23698us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.666146 (+ 259us) write_op.cc:301] APPLY: starting
0110 02:51:27.666158 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.666191 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.666221 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.666272 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.666301 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:27.666307 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:27.666363 (+ 56us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.666427 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.666444 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:27.666449 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.666511 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.666514 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":212,"apply.run_cpu_time_us":359,"apply.run_wall_time_us":359,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":722,"prepare.run_wall_time_us":722,"replication_time_us":23824}]]}
I20250110 02:51:28.132690 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_3018fa039953413a899fe3a942dcbb94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.133170 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3018fa039953413a899fe3a942dcbb94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.106576 1441 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet e848586bec574a529caaf5e498db79cd
I20250110 02:51:28.135478 2224 tablet_replica.cc:331] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.135502 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.135664 2224 raft_consensus.cc:2238] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.135830 2224 raft_consensus.cc:2267] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.135844 1441 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 109) took 2939 ms. Trace:
I20250110 02:51:28.135897 881 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6b2654f86e4686b8a3034de02d11bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.136013 2549 raft_consensus.cc:491] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.136130 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.136109 2549 raft_consensus.cc:513] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.136281 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d6acae633eb14e5db87b1e36a069422c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.101871 1181 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.109133 2600 raft_consensus.cc:491] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.135962 1441 rpcz_store.cc:276] 0110 02:51:25.196255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.725119 (+2528864us) service_pool.cc:225] Handling call
0110 02:51:27.725598 (+ 479us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:27.725728 (+ 130us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:27.725783 (+ 55us) catalog_manager.cc:2762] Locking tablets
0110 02:51:27.787923 (+ 62140us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:27.795257 (+ 7334us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:27.795331 (+ 74us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:28.106528 (+311197us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:28.135776 (+ 29248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.788692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.788793 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.788799 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:27.788802 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:27.788911 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.788928 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.788934 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:27.789042 (+ 108us) tablet.cc:698] Row locks acquired
0110 02:51:27.789045 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:27.789079 (+ 34us) write_op.cc:270] Start()
0110 02:51:27.789116 (+ 37us) write_op.cc:276] Timestamp: P: 1736477487789022 usec, L: 0
0110 02:51:27.789119 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.789305 (+ 186us) log.cc:844] Serialized 725 byte log entry
0110 02:51:27.793457 (+ 4152us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.793564 (+ 107us) write_op.cc:301] APPLY: starting
0110 02:51:27.793577 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.793620 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.793641 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.793699 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.793736 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:27.793743 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:27.793813 (+ 70us) log.cc:844] Serialized 29 byte log entry
0110 02:51:27.793868 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.793885 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:27.793890 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:27.793954 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.793957 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":695,"raft.queue_time_us":372,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":198,"replication_time_us":4311}]]}
I20250110 02:51:28.136432 2600 raft_consensus.cc:513] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.136669 2549 leader_election.cc:290] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.136719 1441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_428c37996d7b406083d3f703508d1729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.136907 2600 leader_election.cc:290] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.137004 2549 raft_consensus.cc:491] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:28.137092 1441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_428c37996d7b406083d3f703508d1729 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.137090 2549 raft_consensus.cc:513] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.137499 1181 ts_tablet_manager.cc:1397] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.137s user 0.007s sys 0.011s
I20250110 02:51:28.137553 2549 leader_election.cc:290] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.137873 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:28.138414 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e05f69028df49f0afb60b28409b917a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.138365 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.137715 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:28.138844 711 leader_election.cc:343] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.139050 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.139201 764 raft_consensus.cc:2463] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.137948 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e848586bec574a529caaf5e498db79cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:28.139336 709 leader_election.cc:343] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.138298 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.139672 996 leader_election.cc:304] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:28.139891 709 leader_election.cc:343] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.139982 709 leader_election.cc:304] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.139982 2600 raft_consensus.cc:2798] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.140049 2600 raft_consensus.cc:491] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.140111 2600 raft_consensus.cc:3054] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.140252 2549 raft_consensus.cc:2743] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.142230 2600 raft_consensus.cc:513] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.143067 2224 ts_tablet_manager.cc:1905] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.117666 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8f4778347f054b86af697dbe3b287263 (table default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878]) successfully deleted
I20250110 02:51:28.145227 754 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6b2654f86e4686b8a3034de02d11bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.145632 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.145893 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.146100 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6b2654f86e4686b8a3034de02d11bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.146795 2377 raft_consensus.cc:491] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98336b684b41407da8a1949e30cd82e0)
I20250110 02:51:28.146862 1184 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.146986 1184 ts_tablet_manager.cc:1397] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.002s
I20250110 02:51:28.146951 2377 raft_consensus.cc:513] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.147034 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.147459 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.147491 2377 leader_election.cc:290] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.147607 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet d6acae633eb14e5db87b1e36a069422c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.147967 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d6acae633eb14e5db87b1e36a069422c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:28.148320 853 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:28.148900 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.149067 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet e848586bec574a529caaf5e498db79cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.149261 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e848586bec574a529caaf5e498db79cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:28.149571 994 leader_election.cc:343] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.149838 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 04e05a7554a14805bdcbd1824a5bd9d5 (table default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]) successfully deleted
I20250110 02:51:28.150767 917 raft_consensus.cc:3054] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.153028 2478 raft_consensus.cc:491] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.154006 2377 raft_consensus.cc:491] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.154134 2377 raft_consensus.cc:513] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.154685 2377 leader_election.cc:290] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:28.155787 853 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
W20250110 02:51:28.156201 853 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:28.156311 853 leader_election.cc:304] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.156679 2377 raft_consensus.cc:2743] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.160617 1449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 119) took 2680 ms. Trace:
I20250110 02:51:28.160739 1449 rpcz_store.cc:276] 0110 02:51:25.480005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.130280 (+2650275us) service_pool.cc:225] Handling call
0110 02:51:28.130944 (+ 664us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.131329 (+ 385us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.160321 (+ 28992us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.160450 (+ 129us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.160541 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.140991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.141064 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.141068 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:28.141070 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.141142 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.141150 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.141154 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.141206 (+ 52us) tablet.cc:698] Row locks acquired
0110 02:51:28.141208 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.141233 (+ 25us) write_op.cc:270] Start()
0110 02:51:28.141256 (+ 23us) write_op.cc:276] Timestamp: P: 1736477488141189 usec, L: 0
0110 02:51:28.141259 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.141382 (+ 123us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.159118 (+ 17736us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.159873 (+ 755us) write_op.cc:301] APPLY: starting
0110 02:51:28.159884 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.159918 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.159951 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.160042 (+ 91us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.160071 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.160081 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.160130 (+ 49us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.160171 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.160185 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:28.160192 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":704,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":391,"num_ops":2,"prepare.queue_time_us":9305,"prepare.run_cpu_time_us":432,"prepare.run_wall_time_us":431,"replication_time_us":17848,"threads_started":1}]]}
I20250110 02:51:28.161482 1449 master_service.cc:946] ReplaceTablet: received request to replace tablet 2f0f754966c04074a568c05063591990 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.162096 1448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 118) took 2700 ms. Trace:
I20250110 02:51:28.162457 2732 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:28.162209 1448 rpcz_store.cc:276] 0110 02:51:25.461378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.127624 (+2666246us) service_pool.cc:225] Handling call
0110 02:51:28.128379 (+ 755us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.128854 (+ 475us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.161906 (+ 33052us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.161994 (+ 88us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.162061 (+ 67us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.140426 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.140510 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.140516 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.140518 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.140597 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.140607 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.140611 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.140676 (+ 65us) tablet.cc:698] Row locks acquired
0110 02:51:28.140680 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.140708 (+ 28us) write_op.cc:270] Start()
0110 02:51:28.140750 (+ 42us) write_op.cc:276] Timestamp: P: 1736477488140663 usec, L: 0
0110 02:51:28.140754 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.140898 (+ 144us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.159068 (+ 18170us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.159328 (+ 260us) write_op.cc:301] APPLY: starting
0110 02:51:28.159344 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.159393 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.159421 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.159525 (+ 104us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.159563 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.159573 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.159636 (+ 63us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.159689 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.159706 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:28.159713 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.159782 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.159787 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":446,"apply.run_wall_time_us":446,"num_ops":2,"prepare.queue_time_us":11146,"prepare.run_cpu_time_us":529,"prepare.run_wall_time_us":528,"replication_time_us":18289}]]}
I20250110 02:51:28.162744 1448 master_service.cc:946] ReplaceTablet: received request to replace tablet 4de67338a26f4febb5dd7611a72b4723 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.162685 1184 raft_consensus.cc:357] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.162849 1184 raft_consensus.cc:383] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.162889 1184 raft_consensus.cc:738] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
W20250110 02:51:28.166628 853 leader_election.cc:343] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:28.166738 853 leader_election.cc:304] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.166975 2377 raft_consensus.cc:2743] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:28.172142 2377 consensus_queue.cc:1035] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.174513 1184 consensus_queue.cc:260] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.174845 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ced3e3687f44570880e92ecec188c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.176282 2456 consensus_queue.cc:1035] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.153126 2478 raft_consensus.cc:513] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.177009 2478 leader_election.cc:290] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.179953 2352 ts_tablet_manager.cc:1918] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.188740 2600 leader_election.cc:290] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.189980 2352 log.cc:1198] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:28.191782 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.194933 2352 ts_tablet_manager.cc:1939] T 9e72ed2d31a8467f8c092b48d593c1cc P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.195636 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:28.198777 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:28.199236 994 leader_election.cc:343] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.201655 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 120) took 2719 ms. Trace:
I20250110 02:51:28.201782 1456 rpcz_store.cc:276] 0110 02:51:25.482239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.132548 (+2650309us) service_pool.cc:225] Handling call
0110 02:51:28.133254 (+ 706us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.133640 (+ 386us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.201419 (+ 67779us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.201534 (+ 115us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.201618 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.141466 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.141547 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.141553 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.141555 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.141628 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.141638 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.141645 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.141736 (+ 91us) tablet.cc:698] Row locks acquired
0110 02:51:28.141740 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.141770 (+ 30us) write_op.cc:270] Start()
0110 02:51:28.141797 (+ 27us) write_op.cc:276] Timestamp: P: 1736477488141724 usec, L: 0
0110 02:51:28.141802 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.141964 (+ 162us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.200213 (+ 58249us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.200459 (+ 246us) write_op.cc:301] APPLY: starting
0110 02:51:28.200472 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.200516 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.200545 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.200640 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.200686 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.200697 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:28.200765 (+ 68us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.200821 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.200836 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:28.200841 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.200923 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.200926 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":452,"num_ops":2,"prepare.queue_time_us":7317,"prepare.run_cpu_time_us":556,"prepare.run_wall_time_us":555,"replication_time_us":58382}]]}
I20250110 02:51:28.202425 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 7c2c7a313ba94e5ea85e45276658ba04 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.202172 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 165) took 3932 ms. Trace:
I20250110 02:51:28.202983 1441 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 121) took 2682 ms. Trace:
W20250110 02:51:28.203004 994 leader_election.cc:343] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.203060 1441 rpcz_store.cc:276] 0110 02:51:25.520491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.136526 (+2616035us) service_pool.cc:225] Handling call
0110 02:51:28.137197 (+ 671us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.137553 (+ 356us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.202763 (+ 65210us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.202865 (+ 102us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.202937 (+ 72us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.142066 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.142150 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.142156 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.142160 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:28.142237 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.142246 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.142252 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.142319 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:28.142322 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.142346 (+ 24us) write_op.cc:270] Start()
0110 02:51:28.142406 (+ 60us) write_op.cc:276] Timestamp: P: 1736477488142302 usec, L: 0
0110 02:51:28.142410 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.142565 (+ 155us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.200399 (+ 57834us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.200995 (+ 596us) write_op.cc:301] APPLY: starting
0110 02:51:28.201004 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.201033 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.201049 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.201162 (+ 113us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.201193 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.201202 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:28.201271 (+ 69us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.201304 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.201315 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:28.201322 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":557,"apply.run_cpu_time_us":383,"apply.run_wall_time_us":383,"num_ops":2,"prepare.queue_time_us":4165,"prepare.run_cpu_time_us":586,"prepare.run_wall_time_us":586,"replication_time_us":57966}]]}
I20250110 02:51:28.203579 1441 master_service.cc:946] ReplaceTablet: received request to replace tablet 9b937188bf2949af8f54bade4e0ad984 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.202937 2352 rpcz_store.cc:276] 0110 02:51:24.268927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270163 (+ 1236us) service_pool.cc:225] Handling call
0110 02:51:28.086758 (+3816595us) ts_tablet_manager.cc:1145] Deleting tablet 9e72ed2d31a8467f8c092b48d593c1cc
0110 02:51:28.087134 (+ 376us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.131325 (+ 44191us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.179165 (+ 47840us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.201801 (+ 22636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3814329}
I20250110 02:51:28.204831 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 9e72ed2d31a8467f8c092b48d593c1cc (table default.Table_73580b26ac6a47ab81519905e8adfb2f [id=35c7c7fd52294c0299191456fca25950]) successfully deleted
I20250110 02:51:28.205608 2352 tablet_replica.cc:331] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.206351 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 117) took 2746 ms. Trace:
I20250110 02:51:28.206481 1443 rpcz_store.cc:276] 0110 02:51:25.460036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.125904 (+2665868us) service_pool.cc:225] Handling call
0110 02:51:28.138508 (+ 12604us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.138895 (+ 387us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.206152 (+ 67257us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.206246 (+ 94us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.206305 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.142678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.142754 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.142757 (+ 3us) write_op.cc:435] Acquired schema lock
0110 02:51:28.142759 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.143780 (+ 1021us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.143791 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.143798 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.143889 (+ 91us) tablet.cc:698] Row locks acquired
0110 02:51:28.143892 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.143923 (+ 31us) write_op.cc:270] Start()
0110 02:51:28.143951 (+ 28us) write_op.cc:276] Timestamp: P: 1736477488143878 usec, L: 0
0110 02:51:28.143956 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.144099 (+ 143us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.205513 (+ 61414us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.205724 (+ 211us) write_op.cc:301] APPLY: starting
0110 02:51:28.205738 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.205764 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.205791 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.205897 (+ 106us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.205922 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.205928 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.205977 (+ 49us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.206022 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.206033 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:28.206038 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":364,"apply.run_wall_time_us":364,"num_ops":2,"prepare.queue_time_us":3418,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":1806,"replication_time_us":61527}]]}
I20250110 02:51:28.207026 1443 master_service.cc:946] ReplaceTablet: received request to replace tablet 1d634dd0c2044f91b304fe3737330d88 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.207742 2352 raft_consensus.cc:2238] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.208986 2352 raft_consensus.cc:2267] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.213124 2352 ts_tablet_manager.cc:1905] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.217455 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:28.218178 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.218724 1448 catalog_manager.cc:6596] ReplaceTablet: tablet 4de67338a26f4febb5dd7611a72b4723 deleted and replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac
I20250110 02:51:28.218899 1448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 123) took 2536 ms. Trace:
I20250110 02:51:28.219014 1448 rpcz_store.cc:276] 0110 02:51:25.682015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.162687 (+2480672us) service_pool.cc:225] Handling call
0110 02:51:28.209509 (+ 46822us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.218838 (+ 9329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.210990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.211405 (+ 415us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.211411 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.211412 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:28.211502 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.211509 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.211513 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.211585 (+ 72us) tablet.cc:698] Row locks acquired
0110 02:51:28.211587 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.211616 (+ 29us) write_op.cc:270] Start()
0110 02:51:28.211640 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488211563 usec, L: 0
0110 02:51:28.211643 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.211764 (+ 121us) log.cc:844] Serialized 656 byte log entry
0110 02:51:28.215714 (+ 3950us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.216584 (+ 870us) write_op.cc:301] APPLY: starting
0110 02:51:28.216591 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.216615 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.216650 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.216729 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.216785 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.216795 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.216862 (+ 67us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.216897 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.216910 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:28.216916 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.216973 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.216978 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":106995968,"child_traces":[["op",{"apply.queue_time_us":844,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":376,"num_ops":2,"prepare.queue_time_us":584,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":1442,"raft.queue_time_us":293,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":4059}]]}
I20250110 02:51:28.219834 1448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_56ef844e399040a3ac6279fc5d2ed20b" } new_table_name: "default.Table_26326c3a6b1b404bb42e75c5a619be95" modify_external_catalogs: true
I20250110 02:51:28.223212 2691 consensus_queue.cc:1035] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.224275 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.216557 1181 raft_consensus.cc:357] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.226001 1181 raft_consensus.cc:383] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.226074 1181 raft_consensus.cc:738] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.226253 1181 consensus_queue.cc:260] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.226704 1181 ts_tablet_manager.cc:1428] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20250110 02:51:28.227495 1181 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.229595 2691 raft_consensus.cc:491] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.229746 2691 raft_consensus.cc:513] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.230242 2691 leader_election.cc:290] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:28.230744 1448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eed058931194480ca1a27130bfa8ee32 consensus state has no leader
I20250110 02:51:28.230933 1448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 127) took 1723 ms. Trace:
I20250110 02:51:28.231032 1448 rpcz_store.cc:276] 0110 02:51:26.507558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.219680 (+1712122us) service_pool.cc:225] Handling call
0110 02:51:28.220080 (+ 400us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.220352 (+ 272us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:28.220506 (+ 154us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:28.220572 (+ 66us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:28.230517 (+ 9945us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:28.230888 (+ 371us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.221675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.221758 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.221763 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.221764 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:28.221841 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.221869 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.221875 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.221929 (+ 54us) tablet.cc:698] Row locks acquired
0110 02:51:28.221932 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.221957 (+ 25us) write_op.cc:270] Start()
0110 02:51:28.221980 (+ 23us) write_op.cc:276] Timestamp: P: 1736477488221906 usec, L: 0
0110 02:51:28.221984 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.222111 (+ 127us) log.cc:844] Serialized 405 byte log entry
0110 02:51:28.229168 (+ 7057us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.229279 (+ 111us) write_op.cc:301] APPLY: starting
0110 02:51:28.229295 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.229344 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.229404 (+ 60us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.229447 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.229478 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.229488 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.229556 (+ 68us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.229606 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.229621 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:28.229628 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.229712 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.229718 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":423,"num_ops":1,"prepare.queue_time_us":724,"prepare.run_cpu_time_us":516,"prepare.run_wall_time_us":750,"replication_time_us":7152}]]}
W20250110 02:51:28.231535 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
W20250110 02:51:28.231729 853 leader_election.cc:343] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 89f6797b1076468a93fb0b510c553bed
I20250110 02:51:28.231843 853 leader_election.cc:304] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.231806 1448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_5c647fe509fd45eebc5796422650b5d8" } new_table_name: "default.Table_b127feb5603541859f25e8a7b3703c7d" modify_external_catalogs: true
I20250110 02:51:28.232878 2691 raft_consensus.cc:2743] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.237716 1181 tablet_bootstrap.cc:654] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.239316 2456 raft_consensus.cc:491] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.239432 2456 raft_consensus.cc:513] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.239972 1448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfd120d8edf14151b7ebed3b74e2b399 consensus state has no leader
I20250110 02:51:28.240005 2456 leader_election.cc:290] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.240161 1448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 128) took 1119 ms. Trace:
I20250110 02:51:28.240461 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.240260 1448 rpcz_store.cc:276] 0110 02:51:27.120370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.231649 (+1111279us) service_pool.cc:225] Handling call
0110 02:51:28.231961 (+ 312us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.232144 (+ 183us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:28.232279 (+ 135us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:28.232346 (+ 67us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:28.239763 (+ 7417us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:28.240116 (+ 353us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.233108 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.233197 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.233205 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:28.233208 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.233311 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.233325 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.233332 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.233393 (+ 61us) tablet.cc:698] Row locks acquired
0110 02:51:28.233397 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.233428 (+ 31us) write_op.cc:270] Start()
0110 02:51:28.233457 (+ 29us) write_op.cc:276] Timestamp: P: 1736477488233373 usec, L: 0
0110 02:51:28.233463 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.233647 (+ 184us) log.cc:844] Serialized 405 byte log entry
0110 02:51:28.238590 (+ 4943us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.238708 (+ 118us) write_op.cc:301] APPLY: starting
0110 02:51:28.238722 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.238754 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.238790 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.238831 (+ 41us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.238877 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.238883 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.238946 (+ 63us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.238998 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.239015 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:28.239022 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.239090 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.239097 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":389,"num_ops":1,"prepare.queue_time_us":400,"prepare.run_cpu_time_us":601,"prepare.run_wall_time_us":917,"replication_time_us":5098}]]}
W20250110 02:51:28.240849 994 leader_election.cc:343] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.241604 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 116) took 2785 ms. Trace:
I20250110 02:51:28.241771 1457 rpcz_store.cc:276] 0110 02:51:25.456307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.094531 (+2638224us) service_pool.cc:225] Handling call
0110 02:51:28.175425 (+ 80894us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:28.175947 (+ 522us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:28.241377 (+ 65430us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:28.241491 (+ 114us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:28.241570 (+ 79us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.192644 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.192751 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.192757 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.192760 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.192875 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.192885 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.192891 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.192987 (+ 96us) tablet.cc:698] Row locks acquired
0110 02:51:28.192991 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.193031 (+ 40us) write_op.cc:270] Start()
0110 02:51:28.193059 (+ 28us) write_op.cc:276] Timestamp: P: 1736477488192976 usec, L: 0
0110 02:51:28.193063 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.208318 (+ 15255us) log.cc:844] Serialized 433 byte log entry
0110 02:51:28.208596 (+ 278us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:28.215653 (+ 7057us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.216180 (+ 527us) write_op.cc:301] APPLY: starting
0110 02:51:28.216194 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.216225 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.216227 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.216309 (+ 82us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.216335 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:28.216341 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.216396 (+ 55us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.216455 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.216471 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:28.216478 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.216532 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.216536 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":482,"apply.run_cpu_time_us":345,"apply.run_wall_time_us":345,"num_ops":2,"prepare.queue_time_us":16174,"prepare.run_cpu_time_us":1016,"prepare.run_wall_time_us":16016,"raft.queue_time_us":3509,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":222,"replication_time_us":22565,"spinlock_wait_cycles":34712320}]]}
I20250110 02:51:28.247116 1448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_18a8f14e969a458599b59ee194888979" } new_table_name: "default.Table_4e121eab5d694b9f970bacc67c1a3841" modify_external_catalogs: true
I20250110 02:51:28.254680 1457 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_72d32c146dce4aac925f27638bf13312" } new_table_name: "default.Table_6aaaf7ba453a4e74a3a55367388c0584" modify_external_catalogs: true
I20250110 02:51:28.265811 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 7c2c7a313ba94e5ea85e45276658ba04
I20250110 02:51:28.266314 1456 catalog_manager.cc:6596] ReplaceTablet: tablet 7c2c7a313ba94e5ea85e45276658ba04 deleted and replaced by tablet 30645a0642bc43ff957368f928b877d1
I20250110 02:51:28.266445 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 124) took 2481 ms. Trace:
I20250110 02:51:28.266510 1456 rpcz_store.cc:276] 0110 02:51:25.785080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.202352 (+2417272us) service_pool.cc:225] Handling call
0110 02:51:28.210914 (+ 8562us) spinlock_profiling.cc:243] Waited 8.26 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.266382 (+ 55468us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.244425 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.244526 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.244532 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:28.244535 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.244642 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.244654 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.244660 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.244735 (+ 75us) tablet.cc:698] Row locks acquired
0110 02:51:28.244738 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.244767 (+ 29us) write_op.cc:270] Start()
0110 02:51:28.244792 (+ 25us) write_op.cc:276] Timestamp: P: 1736477488244712 usec, L: 0
0110 02:51:28.244797 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.244979 (+ 182us) log.cc:844] Serialized 656 byte log entry
0110 02:51:28.263986 (+ 19007us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.264776 (+ 790us) write_op.cc:301] APPLY: starting
0110 02:51:28.264788 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.264848 (+ 60us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.264952 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.265180 (+ 228us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.265272 (+ 92us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.265280 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:28.265399 (+ 119us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.265480 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.265497 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:28.265504 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":19000064,"child_traces":[["op",{"apply.queue_time_us":687,"num_ops":2,"prepare.queue_time_us":287,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":886,"replication_time_us":19162}]]}
I20250110 02:51:28.267959 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.268846 2352 ts_tablet_manager.cc:1918] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.268998 2352 log.cc:1198] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:28.269403 2352 ts_tablet_manager.cc:1939] T f5f7c6ca2cd747e78c24fe0b0bf99bf4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:28.269903 853 consensus_peers.cc:487] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.267139 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7927766540604688908ced9878eecd08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.270812 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7927766540604688908ced9878eecd08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:28.181388 711 leader_election.cc:343] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.182174 2199 raft_consensus.cc:491] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.274056 711 leader_election.cc:304] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.274108 2199 raft_consensus.cc:513] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.274364 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 92982.9us
I20250110 02:51:28.274586 2632 raft_consensus.cc:2743] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.274685 2199 leader_election.cc:290] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.275065 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.275226 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.212409 2357 ts_tablet_manager.cc:1918] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.221223 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:28.275673 711 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:28.275792 709 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.275887 709 leader_election.cc:304] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.276293 2357 log.cc:1198] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/95b14d7592024a34baf7756d92e546d9
I20250110 02:51:28.276347 2199 raft_consensus.cc:2743] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.276710 2357 ts_tablet_manager.cc:1939] T 95b14d7592024a34baf7756d92e546d9 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.277005 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 121) took 4026 ms. Trace:
I20250110 02:51:28.277084 2357 rpcz_store.cc:276] 0110 02:51:24.250066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.251330 (+ 1264us) service_pool.cc:225] Handling call
0110 02:51:28.118510 (+3867180us) ts_tablet_manager.cc:1145] Deleting tablet 95b14d7592024a34baf7756d92e546d9
0110 02:51:28.118525 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.176924 (+ 58399us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.211629 (+ 34705us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.276993 (+ 65364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3866622}
I20250110 02:51:28.277459 2357 tablet_replica.cc:331] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.277631 2357 raft_consensus.cc:2238] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.277761 2357 raft_consensus.cc:2267] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.278240 2357 ts_tablet_manager.cc:1905] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.278543 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.278935 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.279647 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 95b14d7592024a34baf7756d92e546d9 (table default.Table_bf5730ffdb414255925a1360ec7d7626 [id=8dafaf90f6e7460284b6fc3434eed1e4]) successfully deleted
I20250110 02:51:28.283970 2357 ts_tablet_manager.cc:1918] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:28.284188 2357 log.cc:1198] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:28.284449 2357 ts_tablet_manager.cc:1939] T 80186dd7d1a240068584b64ce9e9b03f P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.284735 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 125) took 4034 ms. Trace:
I20250110 02:51:28.284822 2357 rpcz_store.cc:276] 0110 02:51:24.250539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.251458 (+ 919us) service_pool.cc:225] Handling call
0110 02:51:28.277421 (+4025963us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:28.277429 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.281052 (+ 3623us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.283903 (+ 2851us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.284723 (+ 820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4025814}
I20250110 02:51:28.285238 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 80186dd7d1a240068584b64ce9e9b03f (table default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]) successfully deleted
I20250110 02:51:28.286319 2199 raft_consensus.cc:491] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.286465 2199 raft_consensus.cc:513] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.286724 1181 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.286831 1181 ts_tablet_manager.cc:1397] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.000s
I20250110 02:51:28.286993 2199 leader_election.cc:290] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.287518 1181 raft_consensus.cc:357] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.287671 1181 raft_consensus.cc:383] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.287726 1181 raft_consensus.cc:738] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.287904 1181 consensus_queue.cc:260] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.288208 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.288342 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.288509 1181 ts_tablet_manager.cc:1428] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250110 02:51:28.288714 709 leader_election.cc:343] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.288793 1181 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:28.289222 711 leader_election.cc:343] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.289324 711 leader_election.cc:304] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.290007 2199 raft_consensus.cc:2743] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.290329 1181 tablet_bootstrap.cc:654] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.292075 1188 tablet_bootstrap.cc:492] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.292313 2733 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:28.292641 1188 ts_tablet_manager.cc:1397] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.214s user 0.000s sys 0.023s
I20250110 02:51:28.299204 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 154) took 4031 ms. Trace:
I20250110 02:51:28.276283 1184 ts_tablet_manager.cc:1428] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.129s user 0.002s sys 0.000s
I20250110 02:51:28.300446 2352 rpcz_store.cc:276] 0110 02:51:24.267863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270318 (+ 2455us) service_pool.cc:225] Handling call
0110 02:51:28.205175 (+3934857us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:28.205186 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.261179 (+ 55993us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.268736 (+ 7557us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.299139 (+ 30403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3934430}
I20250110 02:51:28.300758 1184 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.299507 1188 raft_consensus.cc:357] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.300930 1188 raft_consensus.cc:383] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.300909 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]) successfully deleted
I20250110 02:51:28.301014 1188 raft_consensus.cc:738] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.301234 1188 consensus_queue.cc:260] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.301884 1188 ts_tablet_manager.cc:1428] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250110 02:51:28.291752 2737 tablet.cc:1721] T 361c332d66984cf7ab84d74e913a5bde P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250110 02:51:28.296373 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 134117us
I20250110 02:51:28.302681 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.302876 764 raft_consensus.cc:2463] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.303450 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efa857721c34003ad91912ef2d7b381" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.303615 764 raft_consensus.cc:3054] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.305897 764 raft_consensus.cc:2463] T 7efa857721c34003ad91912ef2d7b381 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.309445 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.309578 768 raft_consensus.cc:2463] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.312479 913 raft_consensus.cc:3054] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.313797 1181 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.313977 1181 ts_tablet_manager.cc:1397] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20250110 02:51:28.314625 1181 raft_consensus.cc:357] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.314771 996 leader_election.cc:304] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.314815 1181 raft_consensus.cc:383] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.314937 1181 raft_consensus.cc:738] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.315128 1181 consensus_queue.cc:260] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.315500 1181 ts_tablet_manager.cc:1428] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:28.315752 1181 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.316118 2352 tablet_replica.cc:331] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.316622 2352 raft_consensus.cc:2238] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.317030 2352 raft_consensus.cc:2267] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.317188 1181 tablet_bootstrap.cc:654] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.316426 1188 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.319427 1188 tablet_bootstrap.cc:654] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.320204 1184 tablet_bootstrap.cc:654] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.320544 2352 ts_tablet_manager.cc:1905] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.326217 996 leader_election.cc:304] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.326702 2738 raft_consensus.cc:2798] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.326814 2738 raft_consensus.cc:491] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.326898 2738 raft_consensus.cc:3054] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.329371 2738 raft_consensus.cc:513] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.330224 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:28.330704 994 leader_election.cc:343] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.330964 2738 leader_election.cc:290] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.331099 2738 raft_consensus.cc:2798] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.331449 2738 raft_consensus.cc:695] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.331665 2738 consensus_queue.cc:237] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.335719 2224 ts_tablet_manager.cc:1918] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.335835 2224 log.cc:1198] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f5e88d1bbc304b4ebbc849a7d783052e
I20250110 02:51:28.336151 2224 ts_tablet_manager.cc:1939] T f5e88d1bbc304b4ebbc849a7d783052e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.336794 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 142) took 4768 ms. Trace:
I20250110 02:51:28.336797 1441 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 9b937188bf2949af8f54bade4e0ad984
I20250110 02:51:28.337133 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 116) took 4087 ms. Trace:
I20250110 02:51:28.337240 2357 rpcz_store.cc:276] 0110 02:51:24.249533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.251640 (+ 2107us) service_pool.cc:225] Handling call
0110 02:51:28.337006 (+4085366us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:28.337016 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.337106 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4085261}
I20250110 02:51:28.337347 1441 catalog_manager.cc:6596] ReplaceTablet: tablet 9b937188bf2949af8f54bade4e0ad984 deleted and replaced by tablet 70e53b838be846c7b3ac44dc75aa3730
I20250110 02:51:28.337469 1441 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 125) took 2505 ms. Trace:
I20250110 02:51:28.337631 2357 tablet_replica.cc:331] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.337796 2357 raft_consensus.cc:2238] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.338080 2357 raft_consensus.cc:2267] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.337577 1441 rpcz_store.cc:276] 0110 02:51:25.832386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.203517 (+2371131us) service_pool.cc:225] Handling call
0110 02:51:28.210738 (+ 7221us) spinlock_profiling.cc:243] Waited 6.96 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.337434 (+126696us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.261208 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.261342 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.261349 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:28.261354 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:28.261434 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.261441 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.261445 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.261516 (+ 71us) tablet.cc:698] Row locks acquired
0110 02:51:28.261518 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.261545 (+ 27us) write_op.cc:270] Start()
0110 02:51:28.261566 (+ 21us) write_op.cc:276] Timestamp: P: 1736477488261487 usec, L: 0
0110 02:51:28.261569 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.261700 (+ 131us) log.cc:844] Serialized 622 byte log entry
0110 02:51:28.335142 (+ 73442us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.336204 (+ 1062us) write_op.cc:301] APPLY: starting
0110 02:51:28.336217 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.336252 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.336280 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.336367 (+ 87us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.336409 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.336414 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:28.336457 (+ 43us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.336504 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.336515 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:28.336519 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":16012032,"child_traces":[["op",{"apply.queue_time_us":1029,"apply.run_cpu_time_us":365,"apply.run_wall_time_us":365,"num_ops":2,"prepare.queue_time_us":4027,"prepare.run_cpu_time_us":569,"prepare.run_wall_time_us":872,"replication_time_us":73559,"threads_started":1}]]}
I20250110 02:51:28.338845 2357 ts_tablet_manager.cc:1905] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:28.339180 1457 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09fd9734714a4847ab2a4ebc37af8960 consensus state has no leader
W20250110 02:51:28.339993 1448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6598311f59c6476ba339b8e57961cf67 consensus state has no leader
I20250110 02:51:28.340013 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b937188bf2949af8f54bade4e0ad984 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e53b838be846c7b3ac44dc75aa3730) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.340159 1448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 129) took 1002 ms. Trace:
I20250110 02:51:28.340265 1448 rpcz_store.cc:276] 0110 02:51:27.337902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.246969 (+909067us) service_pool.cc:225] Handling call
0110 02:51:28.247319 (+ 350us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.247503 (+ 184us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:28.247643 (+ 140us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:28.247728 (+ 85us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:28.339814 (+ 92086us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:28.340096 (+ 282us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.248233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.248320 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.248325 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.248327 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.248415 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.248424 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.248428 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.248480 (+ 52us) tablet.cc:698] Row locks acquired
0110 02:51:28.248483 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.248507 (+ 24us) write_op.cc:270] Start()
0110 02:51:28.248532 (+ 25us) write_op.cc:276] Timestamp: P: 1736477488248452 usec, L: 0
0110 02:51:28.248537 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.248675 (+ 138us) log.cc:844] Serialized 405 byte log entry
0110 02:51:28.335020 (+ 86345us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.335289 (+ 269us) write_op.cc:301] APPLY: starting
0110 02:51:28.335308 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.335350 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.335374 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.335419 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.335449 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.335456 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:28.335528 (+ 72us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.335596 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.335622 (+ 26us) write_op.cc:454] Released schema lock
0110 02:51:28.335628 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.335698 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.335701 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":402,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":498,"prepare.run_wall_time_us":818,"replication_time_us":86447}]]}
I20250110 02:51:28.340723 1188 tablet_bootstrap.cc:492] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.340864 1188 ts_tablet_manager.cc:1397] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.007s
I20250110 02:51:28.341673 1188 raft_consensus.cc:357] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.341893 1188 raft_consensus.cc:383] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.341975 1188 raft_consensus.cc:738] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.342214 1188 consensus_queue.cc:260] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.343026 1188 ts_tablet_manager.cc:1428] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:28.343441 1188 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.345108 1188 tablet_bootstrap.cc:654] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:28.347363 1444 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet dc6b2654f86e4686b8a3034de02d11bf consensus state has no leader
I20250110 02:51:28.348987 2378 consensus_queue.cc:1035] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:28.354244 1444 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b5f162de87044005bdafe93292ff71b8 consensus state has no leader
I20250110 02:51:28.357519 883 tablet_service.cc:1514] Processing DeleteTablet for tablet e744cdc1fcb94d46b48b684763ecf1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.357754 886 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.357956 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.358151 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e72ed2d31a8467f8c092b48d593c1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.358315 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 585320c0970c4097b195caaae33789b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.358556 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 with delete_type TABLET_DATA_DELETED (replaced by tablet 7064f3f16f6144a4b624eff2ad3ccf62) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.352375 2456 consensus_queue.cc:1035] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.358563 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 with delete_type TABLET_DATA_DELETED (replaced by tablet 815767a4c7624852bfedab9fa6cb5f44) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.352638 996 leader_election.cc:304] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.359128 2456 raft_consensus.cc:2798] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.359201 2456 raft_consensus.cc:491] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.359298 2456 raft_consensus.cc:3054] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.359448 1181 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.359567 1181 ts_tablet_manager.cc:1397] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250110 02:51:28.360231 1181 raft_consensus.cc:357] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.360422 1181 raft_consensus.cc:383] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.360488 1181 raft_consensus.cc:738] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.360669 1181 consensus_queue.cc:260] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.361121 1181 ts_tablet_manager.cc:1428] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250110 02:51:28.361828 1450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 121) took 1182 ms (client timeout 1000 ms). Trace:
I20250110 02:51:28.362026 2456 raft_consensus.cc:513] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.362250 2352 ts_tablet_manager.cc:1918] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.362430 2352 log.cc:1198] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:28.362730 2456 leader_election.cc:290] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.362931 2352 ts_tablet_manager.cc:1939] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.363502 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 166) took 4094 ms. Trace:
I20250110 02:51:28.364039 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b937188bf2949af8f54bade4e0ad984 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e53b838be846c7b3ac44dc75aa3730) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.364225 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b937188bf2949af8f54bade4e0ad984 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e53b838be846c7b3ac44dc75aa3730) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.364578 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fc9826ef554a5699e365d10d02e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:28.364885 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
I20250110 02:51:28.365196 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb5412b78104d8085878c58b191ab27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.365337 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.365379 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.365549 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ae1c19d2dc4778a1cebddbd505f907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.365716 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.366588 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:28.367274 994 leader_election.cc:343] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.363638 2352 rpcz_store.cc:276] 0110 02:51:24.269032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270356 (+ 1324us) service_pool.cc:225] Handling call
0110 02:51:28.300824 (+4030468us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:28.316028 (+ 15204us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.359592 (+ 43564us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.362136 (+ 2544us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.363456 (+ 1320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4030360}
I20250110 02:51:28.367743 2352 tablet_replica.cc:331] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.367707 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (table default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]) successfully deleted
I20250110 02:51:28.367996 2352 raft_consensus.cc:2238] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.367982 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.368192 768 raft_consensus.cc:3054] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.368391 2352 raft_consensus.cc:2267] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.368902 2352 ts_tablet_manager.cc:1905] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.369273 2378 raft_consensus.cc:491] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.369424 2378 raft_consensus.cc:513] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.369952 2378 leader_election.cc:290] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:28.370851 1444 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 109) took 1679 ms (client timeout 1000 ms). Trace:
W20250110 02:51:28.370981 1444 rpcz_store.cc:269] 0110 02:51:26.691025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.915406 (+224381us) service_pool.cc:225] Handling call
0110 02:51:28.370819 (+1455413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80385,"reported_tablets":270}
I20250110 02:51:28.371146 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.371330 770 raft_consensus.cc:2388] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:28.372042 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6555b0a4491046bf8f5cd4ee406c3955" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.372576 853 leader_election.cc:304] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.372826 2378 raft_consensus.cc:3054] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.372992 2224 rpcz_store.cc:276] 0110 02:51:23.568100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.576565 (+ 8465us) service_pool.cc:225] Handling call
0110 02:51:28.135431 (+4558866us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:28.135441 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.271425 (+135984us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.335631 (+ 64206us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.336780 (+ 1149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4558523}
I20250110 02:51:28.373852 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f5e88d1bbc304b4ebbc849a7d783052e (table default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]) successfully deleted
I20250110 02:51:28.374397 1181 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.374605 2224 tablet_replica.cc:331] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.374807 2224 raft_consensus.cc:2238] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.375200 2224 raft_consensus.cc:2267] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.375316 2378 raft_consensus.cc:2743] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:28.377177 1440 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 129) took 1029 ms (client timeout 1000 ms). Trace:
W20250110 02:51:28.361915 1450 rpcz_store.cc:269] 0110 02:51:27.179594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.509982 (+330388us) service_pool.cc:225] Handling call
0110 02:51:27.769877 (+259895us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.769882 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:28.361775 (+591893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:27.764150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:27.764237 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:27.764242 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:27.764245 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:27.764367 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:27.764377 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:27.764382 (+ 5us) tablet.cc:682] Acquiring locks for 3 operations
0110 02:51:27.764459 (+ 77us) tablet.cc:698] Row locks acquired
0110 02:51:27.764461 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:27.764492 (+ 31us) write_op.cc:270] Start()
0110 02:51:27.764519 (+ 27us) write_op.cc:276] Timestamp: P: 1736477487764438 usec, L: 0
0110 02:51:27.764523 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:27.764693 (+ 170us) log.cc:844] Serialized 1275 byte log entry
0110 02:51:27.767317 (+ 2624us) op_driver.cc:464] REPLICATION: finished
0110 02:51:27.767427 (+ 110us) write_op.cc:301] APPLY: starting
0110 02:51:27.767442 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:27.767509 (+ 67us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:27.767551 (+ 42us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:27.767625 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:27.767656 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0110 02:51:27.767662 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:27.767731 (+ 69us) log.cc:844] Serialized 35 byte log entry
0110 02:51:27.767791 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:27.767810 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:27.767817 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":65367,"reported_tablets":259,"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":446,"apply.run_wall_time_us":445,"num_ops":3,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":642,"raft.queue_time_us":79,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":197,"replication_time_us":2761}]]}
W20250110 02:51:28.382822 1440 rpcz_store.cc:269] 0110 02:51:27.347630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.538469 (+190839us) service_pool.cc:225] Handling call
0110 02:51:27.643516 (+105047us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538df66b8 00007f952fefb716 00007f952fefb445 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.377146 (+733630us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":145859,"reported_tablets":243,"spinlock_wait_cycles":2518912}
I20250110 02:51:28.390681 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.391327 1188 tablet_bootstrap.cc:492] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.392515 1188 ts_tablet_manager.cc:1397] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.007s
I20250110 02:51:28.394677 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.394841 770 raft_consensus.cc:3054] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.396039 1188 raft_consensus.cc:357] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.398675 1188 raft_consensus.cc:383] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.398754 768 raft_consensus.cc:2463] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.399031 1188 raft_consensus.cc:738] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.399513 1188 consensus_queue.cc:260] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.400749 1188 ts_tablet_manager.cc:1428] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:28.401324 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:28.401692 1188 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.403586 2378 consensus_queue.cc:1035] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.407912 1441 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_0de0a6680c0f4531a75432a618576660" } new_table_name: "default.Table_3e90520ccabf4d1d9f02e80796522910" modify_external_catalogs: true
I20250110 02:51:28.409420 1448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_2d3082a17dfd4f50b025f2da061be352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.409922 1448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d3082a17dfd4f50b025f2da061be352 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.411015 1456 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_7d36d6648c3f4c45b1672434450cc64b" } modify_external_catalogs: true
I20250110 02:51:28.411527 1456 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_7d36d6648c3f4c45b1672434450cc64b" } modify_external_catalogs: true
I20250110 02:51:28.411173 1444 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_acfdadc9edbf4eaf94f0a667273d93ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.413290 1444 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acfdadc9edbf4eaf94f0a667273d93ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.412052 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_2433934279a049aca30f6cccd6c71952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.414038 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2433934279a049aca30f6cccd6c71952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.415266 1439 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_46ddbd83b65347bba213571663209760 [id=edc4fd777f664491be5d464c39378878] alter complete (version 1)
I20250110 02:51:28.416568 2224 ts_tablet_manager.cc:1905] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.419660 1181 tablet_bootstrap.cc:654] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.423204 2352 ts_tablet_manager.cc:1918] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.423266 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.423388 2352 log.cc:1198] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f14f06ba7d0f44418693e36f6bc6d888
I20250110 02:51:28.423739 2352 ts_tablet_manager.cc:1939] T f14f06ba7d0f44418693e36f6bc6d888 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.423918 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4778347f054b86af697dbe3b287263 with delete_type TABLET_DATA_DELETED (replaced by tablet abbd43638a2c47dcba4f6f90e8676f42) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.424101 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.425098 2289 consensus_queue.cc:1035] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:28.425634 853 consensus_peers.cc:487] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.426645 2549 raft_consensus.cc:491] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:28.427570 853 consensus_peers.cc:487] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.428197 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4d3270e3a04cc99f90943b9573ec7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.428357 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.428463 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.428607 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.428702 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b14d7592024a34baf7756d92e546d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.426774 2549 raft_consensus.cc:513] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.429731 2549 leader_election.cc:290] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.429787 1450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_5bef6387b90c461e8266e7b3052ee939" } modify_external_catalogs: true
I20250110 02:51:28.429967 1450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_5bef6387b90c461e8266e7b3052ee939" } modify_external_catalogs: true
I20250110 02:51:28.431798 1440 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_add5accf5d07401eadb673123fda5ae6" } modify_external_catalogs: true
I20250110 02:51:28.432014 1440 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_add5accf5d07401eadb673123fda5ae6" } modify_external_catalogs: true
I20250110 02:51:28.432533 770 raft_consensus.cc:2463] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.432802 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:28.433708 709 leader_election.cc:343] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.435189 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.436081 996 leader_election.cc:304] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.436373 2736 raft_consensus.cc:2798] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.436607 2736 raft_consensus.cc:695] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.436904 2736 consensus_queue.cc:237] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.440627 711 leader_election.cc:343] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.440725 711 leader_election.cc:304] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.441056 2549 raft_consensus.cc:2743] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.450682 2377 consensus_queue.cc:1035] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.452224 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 157) took 4184 ms. Trace:
I20250110 02:51:28.452293 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:28.452342 2352 rpcz_store.cc:276] 0110 02:51:24.268126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270453 (+ 2327us) service_pool.cc:225] Handling call
0110 02:51:28.367686 (+4097233us) ts_tablet_manager.cc:1145] Deleting tablet f14f06ba7d0f44418693e36f6bc6d888
0110 02:51:28.367696 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.377776 (+ 10080us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.423082 (+ 45306us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.452132 (+ 29050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4097129}
I20250110 02:51:28.452754 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f14f06ba7d0f44418693e36f6bc6d888 (table default.Table_7cebc36aba3d43e0877447c797c945f7 [id=e4afa8bc8a2d444aba73032b1b3de135]) successfully deleted
I20250110 02:51:28.453279 2352 tablet_replica.cc:331] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.457386 2549 consensus_queue.cc:1035] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.458174 2549 raft_consensus.cc:491] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.458288 2549 raft_consensus.cc:513] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.453491 2352 raft_consensus.cc:2238] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.458897 2549 leader_election.cc:290] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.458961 2352 raft_consensus.cc:2267] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.459439 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e554d00d05b4fa7a14c9b6159f9d726" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.459605 2352 ts_tablet_manager.cc:1905] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:28.459937 709 leader_election.cc:343] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.460534 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e554d00d05b4fa7a14c9b6159f9d726" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:28.461416 711 leader_election.cc:343] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.461529 711 leader_election.cc:304] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.461899 2549 raft_consensus.cc:2743] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:28.462643 853 consensus_peers.cc:487] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.470667 2352 ts_tablet_manager.cc:1918] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.475734 996 leader_election.cc:304] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.476025 2478 consensus_queue.cc:1035] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.476399 2478 raft_consensus.cc:2798] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.476569 2478 raft_consensus.cc:695] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.476593 2600 raft_consensus.cc:491] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.476686 2600 raft_consensus.cc:513] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.476790 2478 consensus_queue.cc:237] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.477068 2735 raft_consensus.cc:491] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.477221 2735 raft_consensus.cc:513] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.477298 2600 leader_election.cc:290] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.477711 2735 leader_election.cc:290] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:28.478428 1441 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc1901385a534e5dad20954b763329a0 consensus state has no leader
I20250110 02:51:28.479308 1443 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 1d634dd0c2044f91b304fe3737330d88
I20250110 02:51:28.483320 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d634dd0c2044f91b304fe3737330d88 with delete_type TABLET_DATA_DELETED (replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.484443 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d634dd0c2044f91b304fe3737330d88 with delete_type TABLET_DATA_DELETED (replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.484967 1443 catalog_manager.cc:6596] ReplaceTablet: tablet 1d634dd0c2044f91b304fe3737330d88 deleted and replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf
I20250110 02:51:28.485039 2549 raft_consensus.cc:491] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.485086 1443 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 126) took 2333 ms. Trace:
I20250110 02:51:28.485165 2549 raft_consensus.cc:513] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.485749 2549 leader_election.cc:290] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.490748 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.490695 1443 rpcz_store.cc:276] 0110 02:51:26.151818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.206960 (+2055142us) service_pool.cc:225] Handling call
0110 02:51:28.210567 (+ 3607us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.485059 (+274492us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.391601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.391689 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.391694 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.391697 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.391784 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.391791 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.391795 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.391858 (+ 63us) tablet.cc:698] Row locks acquired
0110 02:51:28.391860 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.391889 (+ 29us) write_op.cc:270] Start()
0110 02:51:28.391913 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488391820 usec, L: 0
0110 02:51:28.391916 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.415187 (+ 23271us) log.cc:844] Serialized 622 byte log entry
0110 02:51:28.415685 (+ 498us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:28.473087 (+ 57402us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.473404 (+ 317us) write_op.cc:301] APPLY: starting
0110 02:51:28.473418 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.473461 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.473463 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.473552 (+ 89us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.478414 (+ 4862us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.478424 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.478501 (+ 77us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.478561 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.478578 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:28.478587 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.478655 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.478658 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":7639936,"child_traces":[["op",{"apply.queue_time_us":266,"apply.run_cpu_time_us":491,"apply.run_wall_time_us":5243,"num_ops":2,"prepare.queue_time_us":10729,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":24120,"replication_time_us":81142,"spinlock_wait_cycles":53133056}]]}
I20250110 02:51:28.491768 1449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 2f0f754966c04074a568c05063591990
I20250110 02:51:28.492507 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0f754966c04074a568c05063591990 with delete_type TABLET_DATA_DELETED (replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.492609 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0f754966c04074a568c05063591990 with delete_type TABLET_DATA_DELETED (replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:28.492748 711 leader_election.cc:343] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.502808 1449 catalog_manager.cc:6596] ReplaceTablet: tablet 2f0f754966c04074a568c05063591990 deleted and replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6
I20250110 02:51:28.502943 1449 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 122) took 2849 ms. Trace:
I20250110 02:51:28.503196 1449 rpcz_store.cc:276] 0110 02:51:25.653660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.161404 (+2507744us) service_pool.cc:225] Handling call
0110 02:51:28.209293 (+ 47889us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538e080ab 00007f952ff0ab77 00007f952ff0a8a5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.502913 (+293620us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.415759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.415879 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.415884 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.415901 (+ 17us) tablet.cc:659] Decoding operations
0110 02:51:28.416011 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.416023 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.416029 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:28.416183 (+ 154us) tablet.cc:698] Row locks acquired
0110 02:51:28.416186 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.416223 (+ 37us) write_op.cc:270] Start()
0110 02:51:28.416253 (+ 30us) write_op.cc:276] Timestamp: P: 1736477488416147 usec, L: 0
0110 02:51:28.416257 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.416446 (+ 189us) log.cc:844] Serialized 622 byte log entry
0110 02:51:28.473160 (+ 56714us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.473621 (+ 461us) write_op.cc:301] APPLY: starting
0110 02:51:28.473631 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.473676 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.474529 (+ 853us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.474624 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.474671 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.474677 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.474743 (+ 66us) log.cc:844] Serialized 29 byte log entry
0110 02:51:28.474800 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.474818 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:28.474825 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.474914 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.474920 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":109045504,"child_traces":[["op",{"apply.queue_time_us":419,"apply.run_cpu_time_us":490,"apply.run_wall_time_us":1279,"num_ops":2,"prepare.queue_time_us":34255,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":1167,"replication_time_us":56884}]]}
I20250110 02:51:28.509110 2549 raft_consensus.cc:491] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.509299 2549 raft_consensus.cc:513] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.510068 2549 leader_election.cc:290] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.510923 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:28.511428 711 leader_election.cc:343] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.522262 2549 consensus_queue.cc:1035] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.539989 1184 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.540198 1184 ts_tablet_manager.cc:1397] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.239s user 0.002s sys 0.004s
I20250110 02:51:28.540875 1184 raft_consensus.cc:357] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.541082 1184 raft_consensus.cc:383] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.541150 1184 raft_consensus.cc:738] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.541358 1184 consensus_queue.cc:260] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.542001 1184 ts_tablet_manager.cc:1428] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:28.547250 2357 ts_tablet_manager.cc:1918] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.478497 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b2aa3a302a47d3a3cb4dfc2e330cfa" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.554200 2352 log.cc:1198] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f5e88d1bbc304b4ebbc849a7d783052e
I20250110 02:51:28.482769 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d634dd0c2044f91b304fe3737330d88 with delete_type TABLET_DATA_DELETED (replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.554572 2352 ts_tablet_manager.cc:1939] T f5e88d1bbc304b4ebbc849a7d783052e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:28.554545 994 leader_election.cc:343] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.486390 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.492527 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0f754966c04074a568c05063591990 with delete_type TABLET_DATA_DELETED (replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:28.496959 853 consensus_peers.cc:487] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:28.562244 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 65708.3us
I20250110 02:51:28.563417 2736 raft_consensus.cc:491] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.563588 2736 raft_consensus.cc:513] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.564229 2736 leader_election.cc:290] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.565289 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 173) took 4295 ms. Trace:
I20250110 02:51:28.565382 2352 rpcz_store.cc:276] 0110 02:51:24.269796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270499 (+ 703us) service_pool.cc:225] Handling call
0110 02:51:28.453185 (+4182686us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:28.453196 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.467494 (+ 14298us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.470545 (+ 3051us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.565260 (+ 94715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4182589}
I20250110 02:51:28.500285 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.510466 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:28.566193 996 leader_election.cc:343] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:28.566322 996 leader_election.cc:304] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.566902 2478 raft_consensus.cc:2743] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.486037 1188 tablet_bootstrap.cc:654] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:28.567942 994 leader_election.cc:343] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.568434 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.567101 2352 tablet_replica.cc:331] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:28.568956 994 leader_election.cc:343] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.569134 2352 raft_consensus.cc:2238] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:28.569998 1434 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dc1901385a534e5dad20954b763329a0 consensus state has no leader
I20250110 02:51:28.570569 2352 raft_consensus.cc:2267] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.573971 2352 ts_tablet_manager.cc:1905] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.579151 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f5e88d1bbc304b4ebbc849a7d783052e (table default.Table_d133af1dd7834a27b1d3ee0e4e788fe3 [id=a109f40b27044627a35f57b9b4ed0ca2]) successfully deleted
I20250110 02:51:28.583374 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 1d064a7e8ea34f18b42407903daf7197
I20250110 02:51:28.585203 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d064a7e8ea34f18b42407903daf7197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.588052 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet f0d4d6e02b1d4334a9993d320217f1f8
I20250110 02:51:28.590159 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.591078 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 071bbce1706c43b6bb690c7de1ead3fe
I20250110 02:51:28.594362 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d064a7e8ea34f18b42407903daf7197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.594584 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.594733 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.596539 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:28.607102 1188 tablet_bootstrap.cc:492] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.607259 1188 ts_tablet_manager.cc:1397] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.206s user 0.009s sys 0.000s
I20250110 02:51:28.607843 1188 raft_consensus.cc:357] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.608101 1188 raft_consensus.cc:383] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.608202 1188 raft_consensus.cc:738] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.608318 1449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_06192f79203e4dd4bda8cda74e08e7c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.608402 1188 consensus_queue.cc:260] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.608911 2357 log.cc:1198] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2e25f9643ac0476a90c93fb9aeb921a3
W20250110 02:51:28.608975 1449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06192f79203e4dd4bda8cda74e08e7c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.609067 1188 ts_tablet_manager.cc:1428] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:28.609318 1188 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.609293 2357 ts_tablet_manager.cc:1939] T 2e25f9643ac0476a90c93fb9aeb921a3 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.609616 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 113) took 4360 ms. Trace:
I20250110 02:51:28.609688 2357 rpcz_store.cc:276] 0110 02:51:24.249200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.252322 (+ 3122us) service_pool.cc:225] Handling call
0110 02:51:28.337592 (+4085270us) ts_tablet_manager.cc:1145] Deleting tablet 2e25f9643ac0476a90c93fb9aeb921a3
0110 02:51:28.337601 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.544380 (+206779us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.547164 (+ 2784us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.609567 (+ 62403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4085123}
I20250110 02:51:28.610261 2357 tablet_replica.cc:331] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.610519 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2e25f9643ac0476a90c93fb9aeb921a3 (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]) successfully deleted
I20250110 02:51:28.612277 1440 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_68d7c1c889464468adcd9153caef7ecc" } new_table_name: "default.Table_7f95dbc0488e40178420f7e412eac048" modify_external_catalogs: true
I20250110 02:51:28.612780 2224 ts_tablet_manager.cc:1918] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.612877 2224 log.cc:1198] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/940445840db3441598eba57d3ccb8c9c
I20250110 02:51:28.613199 2224 ts_tablet_manager.cc:1939] T 940445840db3441598eba57d3ccb8c9c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.613464 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 151) took 5030 ms. Trace:
I20250110 02:51:28.613474 1443 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_633f75bba62141e9a0075b23b2ec4bc0" } modify_external_catalogs: true
I20250110 02:51:28.613551 2224 rpcz_store.cc:276] 0110 02:51:23.582568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.582607 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:28.374523 (+4791916us) ts_tablet_manager.cc:1145] Deleting tablet 940445840db3441598eba57d3ccb8c9c
0110 02:51:28.374531 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.601899 (+227368us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.612664 (+ 10765us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.613435 (+ 771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4791781}
I20250110 02:51:28.613936 1443 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_633f75bba62141e9a0075b23b2ec4bc0" } modify_external_catalogs: true
I20250110 02:51:28.614464 1444 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_9883b2a0f5e448df83b62a280a1cb915"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.555166 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:28.561630 709 leader_election.cc:343] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.615634 709 leader_election.cc:304] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:28.615993 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 54375.3us
W20250110 02:51:28.616420 709 leader_election.cc:343] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.616568 709 leader_election.cc:304] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.616778 2199 raft_consensus.cc:2743] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.576550 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.616961 2199 raft_consensus.cc:2743] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.617086 768 raft_consensus.cc:2463] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.584781 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d064a7e8ea34f18b42407903daf7197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.589749 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.596026 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:28.605571 1184 tablet_bootstrap.cc:492] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.608747 2549 raft_consensus.cc:491] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.619233 2549 raft_consensus.cc:513] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.619787 774 raft_consensus.cc:2463] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.619843 2549 leader_election.cc:290] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.620187 2377 consensus_queue.cc:1035] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.622597 1188 tablet_bootstrap.cc:654] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.623821 1184 tablet_bootstrap.cc:654] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.627489 2357 raft_consensus.cc:2238] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.627744 2357 raft_consensus.cc:2267] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.628360 2357 ts_tablet_manager.cc:1905] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.631446 1188 tablet_bootstrap.cc:492] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.631575 1188 ts_tablet_manager.cc:1397] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250110 02:51:28.635828 1188 raft_consensus.cc:357] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.636040 1188 raft_consensus.cc:383] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.636114 1188 raft_consensus.cc:738] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.636307 1188 consensus_queue.cc:260] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.621335 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.639035 2224 tablet_replica.cc:331] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.639256 2224 raft_consensus.cc:2238] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.639580 2224 raft_consensus.cc:2267] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.640156 2224 ts_tablet_manager.cc:1905] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.621781 1181 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.644207 1181 ts_tablet_manager.cc:1397] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.270s user 0.000s sys 0.003s
I20250110 02:51:28.644845 1181 raft_consensus.cc:357] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.645033 1181 raft_consensus.cc:383] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.645082 1181 raft_consensus.cc:738] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.645247 1181 consensus_queue.cc:260] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.645718 1181 ts_tablet_manager.cc:1428] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:28.645967 1181 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.647154 1181 tablet_bootstrap.cc:654] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.625311 996 leader_election.cc:304] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.648195 2736 raft_consensus.cc:2798] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.648278 2736 raft_consensus.cc:491] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.648330 2736 raft_consensus.cc:3054] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.650352 2736 raft_consensus.cc:513] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.651892 2736 leader_election.cc:290] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.648216 996 leader_election.cc:304] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.653208 2478 raft_consensus.cc:2798] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.653270 2478 raft_consensus.cc:491] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.653324 2478 raft_consensus.cc:3054] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.654803 1188 ts_tablet_manager.cc:1428] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20250110 02:51:28.655021 1188 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.655244 2478 raft_consensus.cc:513] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.651088 2691 consensus_queue.cc:1035] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.655676 2478 leader_election.cc:290] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.652146 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:28.656180 1188 tablet_bootstrap.cc:654] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:28.656512 994 leader_election.cc:343] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.656824 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:28.657259 994 leader_election.cc:343] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.660580 2478 raft_consensus.cc:491] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.660678 2478 raft_consensus.cc:513] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.663787 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_9ea3073c95ce42d6b347ad6af5eb8f0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.664698 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_77210dde99b74eb8b00f98f106690a9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.665165 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 940445840db3441598eba57d3ccb8c9c (table default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]) successfully deleted
I20250110 02:51:28.667742 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4bd5e8317a4801a8134d58c23b7bee" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.667904 774 raft_consensus.cc:3054] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.668543 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.668699 768 raft_consensus.cc:3054] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.668990 1181 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.669091 1181 ts_tablet_manager.cc:1397] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250110 02:51:28.671108 1184 tablet_bootstrap.cc:492] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:28.671294 768 raft_consensus.cc:2463] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.673853 996 leader_election.cc:304] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.674137 2456 raft_consensus.cc:2798] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.674235 2456 raft_consensus.cc:695] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.674439 2456 consensus_queue.cc:237] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.676787 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea3073c95ce42d6b347ad6af5eb8f0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.678181 1188 tablet_bootstrap.cc:492] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.678438 1188 ts_tablet_manager.cc:1397] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
W20250110 02:51:28.678865 1444 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9883b2a0f5e448df83b62a280a1cb915 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:28.679404 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77210dde99b74eb8b00f98f106690a9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.679337 1188 raft_consensus.cc:357] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.679633 1188 raft_consensus.cc:383] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.679733 1188 raft_consensus.cc:738] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.680050 1188 consensus_queue.cc:260] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.678613 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_26326c3a6b1b404bb42e75c5a619be95" } modify_external_catalogs: true
I20250110 02:51:28.676064 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_e634410b037147489f005fd9486c1a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:28.680545 774 raft_consensus.cc:2463] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:28.680804 1188 ts_tablet_manager.cc:1428] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:28.681142 1188 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:28.681355 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e634410b037147489f005fd9486c1a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.681368 996 leader_election.cc:304] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.681592 2738 raft_consensus.cc:2798] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.681684 2738 raft_consensus.cc:695] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:28.681880 2738 consensus_queue.cc:237] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.680352 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_26326c3a6b1b404bb42e75c5a619be95" } modify_external_catalogs: true
I20250110 02:51:28.683128 2478 leader_election.cc:290] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.684172 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:28.684587 994 leader_election.cc:343] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.685355 1181 raft_consensus.cc:357] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.685963 1181 raft_consensus.cc:383] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.686177 1181 raft_consensus.cc:738] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.686584 1181 consensus_queue.cc:260] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.687438 1181 ts_tablet_manager.cc:1428] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250110 02:51:28.688380 1181 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.690550 1188 tablet_bootstrap.cc:654] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.691912 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.692915 1181 tablet_bootstrap.cc:654] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.695375 2736 consensus_queue.cc:1035] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.696066 2735 raft_consensus.cc:491] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.696199 2735 raft_consensus.cc:513] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.696653 2735 leader_election.cc:290] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.696707 895 tablet_service.cc:1514] Processing DeleteTablet for tablet f14f06ba7d0f44418693e36f6bc6d888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:28.697118 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.697511 895 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:28.697551 994 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.706880 1184 ts_tablet_manager.cc:1397] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20250110 02:51:28.707216 2549 raft_consensus.cc:491] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.707356 2199 raft_consensus.cc:491] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:28.707566 2773 raft_consensus.cc:491] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.707726 2773 raft_consensus.cc:513] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.707839 2549 raft_consensus.cc:513] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.708365 2773 leader_election.cc:290] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.708950 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:28.710095 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 110) took 911 ms (client timeout 1000 ms). Trace:
I20250110 02:51:28.710359 2199 raft_consensus.cc:513] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.710867 2199 leader_election.cc:290] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.711802 2549 leader_election.cc:290] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.713879 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.714079 1057 raft_consensus.cc:2463] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
W20250110 02:51:28.714293 711 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:28.712790 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.712878 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09fd9734714a4847ab2a4ebc37af8960" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.715445 853 leader_election.cc:304] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.715672 2773 raft_consensus.cc:2743] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.715901 711 leader_election.cc:304] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:28.716284 2549 raft_consensus.cc:2798] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.716363 2549 raft_consensus.cc:491] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.716439 2549 raft_consensus.cc:3054] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:28.710705 1439 rpcz_store.cc:269] 0110 02:51:27.797933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.826932 (+ 28999us) service_pool.cc:225] Handling call
0110 02:51:28.709747 (+882815us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.123366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.123490 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.123498 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:28.123502 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:28.123607 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.123623 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.123629 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.123700 (+ 71us) tablet.cc:698] Row locks acquired
0110 02:51:28.123704 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:28.123743 (+ 39us) write_op.cc:270] Start()
0110 02:51:28.123778 (+ 35us) write_op.cc:276] Timestamp: P: 1736477488123696 usec, L: 0
0110 02:51:28.123782 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.123963 (+ 181us) log.cc:844] Serialized 527 byte log entry
0110 02:51:28.152362 (+ 28399us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.152462 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:28.152477 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.152519 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.152563 (+ 44us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.152632 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.152682 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.152693 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:28.152768 (+ 75us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.152823 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.152839 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:28.152845 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.152924 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.152930 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:28.304552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.304644 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.304648 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:28.304650 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.304728 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.304735 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.304739 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.304783 (+ 44us) tablet.cc:698] Row locks acquired
0110 02:51:28.304785 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.304814 (+ 29us) write_op.cc:270] Start()
0110 02:51:28.304838 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488304752 usec, L: 0
0110 02:51:28.304841 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.304994 (+ 153us) log.cc:844] Serialized 356 byte log entry
0110 02:51:28.407584 (+102590us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.411516 (+ 3932us) write_op.cc:301] APPLY: starting
0110 02:51:28.411529 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.411560 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.411585 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.411643 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.411713 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.411723 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:28.411790 (+ 67us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.415088 (+ 3298us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.415111 (+ 23us) write_op.cc:454] Released schema lock
0110 02:51:28.415117 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.435122 (+ 20005us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.435128 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6151,"reported_tablets":262,"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":690,"raft.queue_time_us":60,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"replication_time_us":28552}],["op",{"apply.queue_time_us":3880,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":23579,"num_ops":1,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":524,"prepare.run_wall_time_us":523,"replication_time_us":102729}]]}
I20250110 02:51:28.718007 1188 tablet_bootstrap.cc:492] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.718130 1188 ts_tablet_manager.cc:1397] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.002s
I20250110 02:51:28.718895 2549 raft_consensus.cc:513] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.719486 2549 leader_election.cc:290] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.720628 768 raft_consensus.cc:2463] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.723778 1184 raft_consensus.cc:357] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.724705 1184 raft_consensus.cc:383] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.724916 1184 raft_consensus.cc:738] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.725256 1184 consensus_queue.cc:260] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.726944 1184 ts_tablet_manager.cc:1428] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
W20250110 02:51:28.727618 996 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:28.728824 996 leader_election.cc:304] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.728941 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.729562 2735 raft_consensus.cc:2743] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:28.729964 996 consensus_peers.cc:487] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2163c343a694f5ca04ac9f7610d25e3. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.730767 996 leader_election.cc:304] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.730995 1188 raft_consensus.cc:357] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.731248 1188 raft_consensus.cc:383] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.731333 1188 raft_consensus.cc:738] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.731503 2735 raft_consensus.cc:2798] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.731563 2735 raft_consensus.cc:491] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.731616 2735 raft_consensus.cc:3054] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.731599 1188 consensus_queue.cc:260] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.732017 1440 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 656aa84a130c42768e969b9322fbd8a5 consensus state has no leader
I20250110 02:51:28.732038 1188 ts_tablet_manager.cc:1428] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250110 02:51:28.732942 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet bb595013e809480a814cb9c7aed856e6 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.733843 2735 raft_consensus.cc:513] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.734351 2735 leader_election.cc:290] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.735044 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:28.735200 1188 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:28.735448 994 leader_election.cc:343] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.736097 1181 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.736474 1188 tablet_bootstrap.cc:654] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.736572 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.736838 1181 ts_tablet_manager.cc:1397] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.003s
I20250110 02:51:28.737268 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:28.737313 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:28.737466 1053 raft_consensus.cc:3054] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:28.738868 709 leader_election.cc:343] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.739964 1053 raft_consensus.cc:2463] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:28.740569 1449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_12bdc0cc5a9446e9a86edc133f7e4ce1" } new_table_name: "default.Table_b4223b31a5974abb9725bceb8026a8ae" modify_external_catalogs: true
I20250110 02:51:28.742340 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_1646191c9c32465fad0850bda2dd3367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.743218 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1646191c9c32465fad0850bda2dd3367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.737466 1181 raft_consensus.cc:357] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.744210 1181 raft_consensus.cc:383] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.744284 1181 raft_consensus.cc:738] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.744472 1181 consensus_queue.cc:260] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.744976 1181 ts_tablet_manager.cc:1428] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250110 02:51:28.745211 1181 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.746304 1181 tablet_bootstrap.cc:654] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.758589 1188 tablet_bootstrap.cc:492] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:28.758747 1188 ts_tablet_manager.cc:1397] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250110 02:51:28.759395 1188 raft_consensus.cc:357] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.759581 1188 raft_consensus.cc:383] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.759629 1188 raft_consensus.cc:738] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.759816 1188 consensus_queue.cc:260] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.760233 1188 ts_tablet_manager.cc:1428] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:28.760454 1188 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.761745 1188 tablet_bootstrap.cc:654] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.761808 1450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_5ced3e3687f44570880e92ecec188c5d" } new_table_name: "default.Table_7376d50f0b824f59a76abea37a60a400" modify_external_catalogs: true
I20250110 02:51:28.763144 1444 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_aad85d64a3ff4f87b6c4f767f39de7df" } new_table_name: "default.Table_a99fe640359046aa8ff288db7a687676" modify_external_catalogs: true
I20250110 02:51:28.773141 2691 raft_consensus.cc:491] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.773268 2691 raft_consensus.cc:513] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.773835 2691 leader_election.cc:290] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.774943 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.740163 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.781147 768 raft_consensus.cc:3054] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.740459 711 leader_election.cc:304] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:28.786051 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 45624.4us
I20250110 02:51:28.786094 2777 raft_consensus.cc:2798] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:28.786307 2777 raft_consensus.cc:695] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:28.786615 2224 ts_tablet_manager.cc:1918] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.786703 2224 log.cc:1198] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:28.786556 2777 consensus_queue.cc:237] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.792235 2224 ts_tablet_manager.cc:1939] T eff9b10c419242cf85ef1b708bd5a862 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.792603 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 152) took 5209 ms. Trace:
I20250110 02:51:28.793578 2224 rpcz_store.cc:276] 0110 02:51:23.582927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.582964 (+ 37us) service_pool.cc:225] Handling call
0110 02:51:28.638943 (+5055979us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:28.638978 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.642831 (+ 3853us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.786519 (+143688us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.792587 (+ 6068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5055855}
I20250110 02:51:28.768066 1184 tablet_bootstrap.cc:492] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:28.794714 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet eff9b10c419242cf85ef1b708bd5a862 (table default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]) successfully deleted
I20250110 02:51:28.794924 1181 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.795080 1181 ts_tablet_manager.cc:1397] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.001s
I20250110 02:51:28.795693 1181 raft_consensus.cc:357] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.770912 2357 ts_tablet_manager.cc:1918] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.775131 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca895eca14748c19fdb7d363499b544" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.796104 774 raft_consensus.cc:2388] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:28.796533 853 leader_election.cc:304] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.796645 1184 tablet_bootstrap.cc:654] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.796974 2357 log.cc:1198] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f6df074974974d81a83420e7a3a78070
I20250110 02:51:28.797371 2357 ts_tablet_manager.cc:1939] T f6df074974974d81a83420e7a3a78070 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.798034 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 134) took 4543 ms. Trace:
I20250110 02:51:28.798182 2357 rpcz_store.cc:276] 0110 02:51:24.254342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.254409 (+ 67us) service_pool.cc:225] Handling call
0110 02:51:28.610195 (+4355786us) ts_tablet_manager.cc:1145] Deleting tablet f6df074974974d81a83420e7a3a78070
0110 02:51:28.610204 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.753251 (+143047us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.770793 (+ 17542us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.798018 (+ 27225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355641}
I20250110 02:51:28.798821 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f6df074974974d81a83420e7a3a78070 (table default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]) successfully deleted
I20250110 02:51:28.799185 2357 tablet_replica.cc:331] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.799432 2357 raft_consensus.cc:2238] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.799638 2357 raft_consensus.cc:2267] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.800100 2357 ts_tablet_manager.cc:1905] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.804167 915 raft_consensus.cc:3054] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.804244 2378 raft_consensus.cc:3054] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.795080 2224 tablet_replica.cc:331] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.801765 2735 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.804721 1181 raft_consensus.cc:383] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.804786 1181 raft_consensus.cc:738] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.804813 2735 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.804879 709 leader_election.cc:304] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.804980 1181 consensus_queue.cc:260] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:28.805142 709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 66924.3us
W20250110 02:51:28.805303 709 leader_election.cc:343] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:28.805313 2735 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.805374 709 leader_election.cc:304] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.805399 1181 ts_tablet_manager.cc:1428] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.000s sys 0.000s
I20250110 02:51:28.805611 1181 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:28.805703 709 leader_election.cc:343] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:28.805933 709 leader_election.cc:343] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.805966 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.806447 2199 raft_consensus.cc:2743] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 02:51:28.806483 994 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.806862 1181 tablet_bootstrap.cc:654] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.808781 2378 raft_consensus.cc:2743] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.810532 2224 raft_consensus.cc:2238] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.810977 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:28.811276 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:28.811228 2735 consensus_queue.cc:1035] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.811770 1457 catalog_manager.cc:5526] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:28.812115 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:28.812332 1457 catalog_manager.cc:5526] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:28.812718 1457 catalog_manager.cc:5526] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:28.812984 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:28.813325 1457 catalog_manager.cc:5526] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } }
I20250110 02:51:28.813668 2224 raft_consensus.cc:2267] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.813884 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:28.814191 1457 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_6aaaf7ba453a4e74a3a55367388c0584 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 3 got 0
I20250110 02:51:28.817477 2224 ts_tablet_manager.cc:1905] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.819697 2777 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.819937 2777 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.834568 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_627f1a96679c4cad8d88b188e4d769b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:28.835121 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_627f1a96679c4cad8d88b188e4d769b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:28.839875 1188 tablet_bootstrap.cc:492] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:28.842576 1450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d63ccced9a24eb19dcb8f9b3e7269ee consensus state has no leader
I20250110 02:51:28.842813 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:28.844547 1188 ts_tablet_manager.cc:1397] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20250110 02:51:28.845729 1188 raft_consensus.cc:357] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.847697 2773 consensus_queue.cc:1035] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.849097 895 tablet_service.cc:1467] Processing CreateTablet for tablet 1e35e3efbeaf4422b0c244e26189cda4 (DEFAULT_TABLE table=default.Table_5a788687cf5a4bcf9e6f19e3622f3a56 [id=55dc88d102d94b6695598be482449773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.849648 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e35e3efbeaf4422b0c244e26189cda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.855290 2773 raft_consensus.cc:491] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.855437 2773 raft_consensus.cc:513] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.856104 2773 leader_election.cc:290] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:28.858017 853 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:28.863132 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 1e35e3efbeaf4422b0c244e26189cda4 (DEFAULT_TABLE table=default.Table_5a788687cf5a4bcf9e6f19e3622f3a56 [id=55dc88d102d94b6695598be482449773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.863476 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e35e3efbeaf4422b0c244e26189cda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.864907 1188 raft_consensus.cc:383] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.864995 1188 raft_consensus.cc:738] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.865252 1188 consensus_queue.cc:260] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.865939 1188 ts_tablet_manager.cc:1428] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250110 02:51:28.869274 895 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad4aebdda524e97a1e2ce08b279940c (DEFAULT_TABLE table=default.Table_67fee57b82044957be4b163808d12438 [id=3e70e390dbec431a817abd5222808252]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.869629 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad4aebdda524e97a1e2ce08b279940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.874971 2224 ts_tablet_manager.cc:1918] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.875116 2224 log.cc:1198] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:28.875496 2224 ts_tablet_manager.cc:1939] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.878050 2549 raft_consensus.cc:2743] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.879815 1057 raft_consensus.cc:1270] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 2 FOLLOWER]: Refusing update from remote peer 98336b684b41407da8a1949e30cd82e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250110 02:51:28.881142 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.881343 770 raft_consensus.cc:2463] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:28.881927 2777 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:28.882719 853 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:28.882817 853 leader_election.cc:304] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.883175 2773 raft_consensus.cc:2743] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.882759 756 tablet_service.cc:1467] Processing CreateTablet for tablet 1e35e3efbeaf4422b0c244e26189cda4 (DEFAULT_TABLE table=default.Table_5a788687cf5a4bcf9e6f19e3622f3a56 [id=55dc88d102d94b6695598be482449773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.884080 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:28.885844 2357 ts_tablet_manager.cc:1918] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.885928 2357 log.cc:1198] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:28.886200 2357 ts_tablet_manager.cc:1939] T 3b6d57f3e9a44d178300d3505d504854 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.890240 2773 raft_consensus.cc:491] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.890440 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad4aebdda524e97a1e2ce08b279940c (DEFAULT_TABLE table=default.Table_67fee57b82044957be4b163808d12438 [id=3e70e390dbec431a817abd5222808252]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.890413 2773 raft_consensus.cc:513] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.891116 2773 leader_election.cc:290] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.891820 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c302f606650e41a586719491f47de43c (DEFAULT_TABLE table=default.Table_4cffba7d34a0456f9f600de97d6a1737 [id=63d5cf2bd6884439b4ec7ade71aff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.892167 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c302f606650e41a586719491f47de43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.892553 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.892812 774 raft_consensus.cc:2388] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:28.893436 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4fe51f2e914865a2e8bbaa256b8098" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.895159 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.895274 853 leader_election.cc:304] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:28.895332 1057 raft_consensus.cc:2463] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:28.896179 2773 raft_consensus.cc:3054] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.898639 889 tablet_service.cc:1467] Processing CreateTablet for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda (DEFAULT_TABLE table=default.Table_7c6062aaaae94525b7b305bd8fccc78f [id=5d6d68df2bc4459bb14e0c516cdc58b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.899014 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.899243 748 tablet_service.cc:1467] Processing CreateTablet for tablet c302f606650e41a586719491f47de43c (DEFAULT_TABLE table=default.Table_4cffba7d34a0456f9f600de97d6a1737 [id=63d5cf2bd6884439b4ec7ade71aff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.900269 883 tablet_service.cc:1467] Processing CreateTablet for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.900570 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.901196 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.901496 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.901463 711 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:28.901685 2549 consensus_queue.cc:1035] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 02:51:28.902180 2549 raft_consensus.cc:2798] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.902272 2549 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.902331 2549 raft_consensus.cc:3054] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.902624 740 tablet_service.cc:1467] Processing CreateTablet for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda (DEFAULT_TABLE table=default.Table_7c6062aaaae94525b7b305bd8fccc78f [id=5d6d68df2bc4459bb14e0c516cdc58b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.903193 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad4aebdda524e97a1e2ce08b279940c (DEFAULT_TABLE table=default.Table_67fee57b82044957be4b163808d12438 [id=3e70e390dbec431a817abd5222808252]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.903456 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad4aebdda524e97a1e2ce08b279940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.906589 894 tablet_service.cc:1467] Processing CreateTablet for tablet 78db9d8dc1a04c7a98f74c81513d8fb8 (DEFAULT_TABLE table=default.Table_3018fa039953413a899fe3a942dcbb94 [id=44e00ca5f1c54c94ab6a95ad3b0541ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.906891 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78db9d8dc1a04c7a98f74c81513d8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:28.906605 709 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.900336 2773 raft_consensus.cc:2743] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.910401 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e35e3efbeaf4422b0c244e26189cda4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.911523 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda (DEFAULT_TABLE table=default.Table_7c6062aaaae94525b7b305bd8fccc78f [id=5d6d68df2bc4459bb14e0c516cdc58b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.911523 753 tablet_service.cc:1467] Processing CreateTablet for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac (DEFAULT_TABLE table=default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.911823 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.911837 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.913714 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 131) took 4659 ms. Trace:
I20250110 02:51:28.913779 2357 rpcz_store.cc:276] 0110 02:51:24.254016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.254478 (+ 462us) service_pool.cc:225] Handling call
0110 02:51:28.799144 (+4544666us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:28.799153 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.872256 (+ 73103us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.885762 (+ 13506us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.913701 (+ 27939us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4544494}
I20250110 02:51:28.914022 2357 tablet_replica.cc:331] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.914194 2357 raft_consensus.cc:2238] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.914314 2357 raft_consensus.cc:2267] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.914652 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad4aebdda524e97a1e2ce08b279940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.914933 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 3b6d57f3e9a44d178300d3505d504854 (table default.Table_a77dd8b2a8d1473db5913fcea0cdce3e [id=311571d31aa64160b80172dade5034da]) successfully deleted
I20250110 02:51:28.916265 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c302f606650e41a586719491f47de43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.918021 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b60845b3e074e2bbe0dea1e2b8b9eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.918500 996 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:28.918743 2735 raft_consensus.cc:2798] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:28.918814 2735 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:28.918871 2735 raft_consensus.cc:3054] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:28.919939 2773 raft_consensus.cc:491] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.919935 2738 consensus_queue.cc:1035] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.920173 2773 raft_consensus.cc:513] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.921566 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:28.921919 2773 leader_election.cc:290] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.922405 2735 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.923485 2735 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:28.923447 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:28.923906 994 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.924440 2456 consensus_queue.cc:1035] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.924834 2456 consensus_queue.cc:1035] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:28.927840 895 tablet_service.cc:1467] Processing CreateTablet for tablet c302f606650e41a586719491f47de43c (DEFAULT_TABLE table=default.Table_4cffba7d34a0456f9f600de97d6a1737 [id=63d5cf2bd6884439b4ec7ade71aff588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:28.929205 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c302f606650e41a586719491f47de43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:28.929277 2357 ts_tablet_manager.cc:1905] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:28.929317 709 consensus_peers.cc:487] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27a7885a80414f2193400672aa16a911. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:28.932686 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6479d71a5142436b8c6ea2f76c221b33" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:28.939237 2352 ts_tablet_manager.cc:1918] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:28.942237 2357 ts_tablet_manager.cc:1918] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.942502 2357 log.cc:1198] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:28.942958 2357 ts_tablet_manager.cc:1939] T aa6057de9da04f2c82b8f25c5b4e2fe4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:28.943688 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 133) took 4689 ms. Trace:
I20250110 02:51:28.951982 2456 raft_consensus.cc:491] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.952193 2456 raft_consensus.cc:513] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:28.953859 2456 leader_election.cc:290] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:28.960098 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 153) took 5376 ms. Trace:
I20250110 02:51:28.960742 2224 rpcz_store.cc:276] 0110 02:51:23.583222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.583260 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:28.795046 (+5211786us) ts_tablet_manager.cc:1145] Deleting tablet aa6057de9da04f2c82b8f25c5b4e2fe4
0110 02:51:28.795053 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.824568 (+ 29515us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.874873 (+ 50305us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.960063 (+ 85190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5211655}
I20250110 02:51:28.961403 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
I20250110 02:51:28.961489 2224 tablet_replica.cc:331] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.961644 2224 raft_consensus.cc:2238] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:28.961934 2224 raft_consensus.cc:2267] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:28.962428 2224 ts_tablet_manager.cc:1905] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:28.971928 994 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:28.935679 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:28.947633 2549 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.972031 2199 raft_consensus.cc:491] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:28.972687 2199 raft_consensus.cc:513] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.973001 2549 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.973321 774 raft_consensus.cc:2388] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8899dd1aeb384e4ea4523f5fd225c888 in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:28.974169 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:28.974119 2357 rpcz_store.cc:276] 0110 02:51:24.254219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.254671 (+ 452us) service_pool.cc:225] Handling call
0110 02:51:28.913990 (+4659319us) ts_tablet_manager.cc:1145] Deleting tablet aa6057de9da04f2c82b8f25c5b4e2fe4
0110 02:51:28.913997 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.939398 (+ 25401us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.942167 (+ 2769us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.943670 (+ 1503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4659143}
I20250110 02:51:28.974474 1061 raft_consensus.cc:2388] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 98336b684b41407da8a1949e30cd82e0 in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:28.974563 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet aa6057de9da04f2c82b8f25c5b4e2fe4 (table default.Table_cce012e47fbc46f4807cb0cd9adc5568 [id=195eac29a34148d289e59396775f2d0a]) successfully deleted
I20250110 02:51:28.974772 2199 leader_election.cc:290] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:28.975519 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:28.975973 711 leader_election.cc:343] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.976315 2357 tablet_replica.cc:331] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:28.976758 1181 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:28.976913 1181 ts_tablet_manager.cc:1397] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.171s user 0.051s sys 0.039s
I20250110 02:51:28.952754 1188 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:28.980599 2352 log.cc:1198] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8940c201aa104b0b8bc8d866859fe8ab
I20250110 02:51:28.981191 2352 ts_tablet_manager.cc:1939] T 8940c201aa104b0b8bc8d866859fe8ab P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:28.982407 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 156) took 4714 ms. Trace:
I20250110 02:51:28.982570 2352 rpcz_store.cc:276] 0110 02:51:24.268035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270575 (+ 2540us) service_pool.cc:225] Handling call
0110 02:51:28.567061 (+4296486us) ts_tablet_manager.cc:1145] Deleting tablet 8940c201aa104b0b8bc8d866859fe8ab
0110 02:51:28.567070 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.660297 (+ 93227us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.939149 (+278852us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.982362 (+ 43213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4296397}
I20250110 02:51:28.983217 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8940c201aa104b0b8bc8d866859fe8ab (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:28.983587 2352 tablet_replica.cc:331] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:28.973395 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:28.974341 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:28.984421 709 leader_election.cc:343] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.984524 709 leader_election.cc:304] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.984522 1181 raft_consensus.cc:357] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.984695 1181 raft_consensus.cc:383] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:28.984745 1181 raft_consensus.cc:738] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:28.984947 1181 consensus_queue.cc:260] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:28.985045 2357 raft_consensus.cc:2238] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.985198 2357 raft_consensus.cc:2267] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:28.985270 709 leader_election.cc:343] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:28.985391 709 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:28.985616 1181 ts_tablet_manager.cc:1428] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250110 02:51:28.985783 2357 ts_tablet_manager.cc:1905] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:28.986097 1181 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:28.985836 2199 raft_consensus.cc:2743] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.986038 2777 raft_consensus.cc:2743] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 02:51:28.987655 1181 tablet_bootstrap.cc:654] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:28.991340 2224 ts_tablet_manager.cc:1918] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:28.993336 2224 log.cc:1198] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f6df074974974d81a83420e7a3a78070
I20250110 02:51:28.993803 2224 ts_tablet_manager.cc:1939] T f6df074974974d81a83420e7a3a78070 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:28.997227 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 154) took 5413 ms. Trace:
I20250110 02:51:28.997661 2224 rpcz_store.cc:276] 0110 02:51:23.583585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.583621 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:28.961446 (+5377825us) ts_tablet_manager.cc:1145] Deleting tablet f6df074974974d81a83420e7a3a78070
0110 02:51:28.961454 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:28.988476 (+ 27022us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.991266 (+ 2790us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:28.997212 (+ 5946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5377708}
I20250110 02:51:28.998317 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f6df074974974d81a83420e7a3a78070 (table default.Table_b7a2bc15b1114d39944fadd733f2eb4d [id=e07430516ddb47d9bd1c9397946ab300]) successfully deleted
I20250110 02:51:28.998767 2224 tablet_replica.cc:331] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:28.999238 2224 raft_consensus.cc:2238] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:28.999766 2224 raft_consensus.cc:2267] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.000552 1188 tablet_bootstrap.cc:654] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.000888 2224 ts_tablet_manager.cc:1905] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.008720 2357 ts_tablet_manager.cc:1918] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.016980 1188 tablet_bootstrap.cc:492] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.017254 1188 ts_tablet_manager.cc:1397] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.002s
I20250110 02:51:29.018590 1188 raft_consensus.cc:357] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.019009 1188 raft_consensus.cc:383] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.019174 1188 raft_consensus.cc:738] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.019634 1188 consensus_queue.cc:260] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.020555 1188 ts_tablet_manager.cc:1428] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:29.021982 2224 ts_tablet_manager.cc:1918] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.022286 2224 log.cc:1198] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:29.022701 2224 ts_tablet_manager.cc:1939] T b3ae68e4f0254ef38aafd5a1f8f88a54 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.023311 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 155) took 5438 ms. Trace:
I20250110 02:51:29.023730 2224 rpcz_store.cc:276] 0110 02:51:23.584711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.584757 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:28.998689 (+5413932us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:28.998736 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.004161 (+ 5425us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.021900 (+ 17739us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.023297 (+ 1397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5413754}
I20250110 02:51:29.024498 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b3ae68e4f0254ef38aafd5a1f8f88a54 (table default.Table_7881112a09664881843ffe986bb6aac2 [id=de7d56b98072453f8bf1a0eb9e41f1f5]) successfully deleted
I20250110 02:51:29.024914 2224 tablet_replica.cc:331] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.025290 2224 raft_consensus.cc:2238] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.025673 2224 raft_consensus.cc:2267] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.026785 2224 ts_tablet_manager.cc:1905] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.011044 1184 tablet_bootstrap.cc:492] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.037691 1184 ts_tablet_manager.cc:1397] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.270s user 0.002s sys 0.002s
I20250110 02:51:29.018898 770 raft_consensus.cc:2388] T 6479d71a5142436b8c6ea2f76c221b33 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:29.039119 1184 raft_consensus.cc:357] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.039789 1184 raft_consensus.cc:383] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.039914 1184 raft_consensus.cc:738] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.040199 1184 consensus_queue.cc:260] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.040750 1184 ts_tablet_manager.cc:1428] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:29.041033 1184 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.042515 1184 tablet_bootstrap.cc:654] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.043285 853 leader_election.cc:304] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.043686 2224 ts_tablet_manager.cc:1918] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250110 02:51:29.043922 2357 log.cc:1198] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:29.044297 2357 ts_tablet_manager.cc:1939] T eff9b10c419242cf85ef1b708bd5a862 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.044615 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 132) took 4790 ms. Trace:
I20250110 02:51:29.044718 2357 rpcz_store.cc:276] 0110 02:51:24.254098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.254878 (+ 780us) service_pool.cc:225] Handling call
0110 02:51:28.976273 (+4721395us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:28.976280 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.003910 (+ 27630us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.008593 (+ 4683us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.044597 (+ 36004us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7077,"tablet-delete.queue_time_us":4721295}
I20250110 02:51:29.045162 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet eff9b10c419242cf85ef1b708bd5a862 (table default.Table_6ffc0117db5547039d28e581cd91678b [id=8e152142ce6a44bdaa471e5a736dc8b7]) successfully deleted
I20250110 02:51:29.045385 2357 tablet_replica.cc:331] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.045210 2378 raft_consensus.cc:3054] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.046880 2224 log.cc:1198] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/27a7885a80414f2193400672aa16a911
I20250110 02:51:29.047384 2224 ts_tablet_manager.cc:1939] T 27a7885a80414f2193400672aa16a911 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.048015 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 157) took 5462 ms. Trace:
I20250110 02:51:29.048426 2224 rpcz_store.cc:276] 0110 02:51:23.585230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.585270 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:29.024870 (+5439600us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:29.024879 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.030182 (+ 5303us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.043583 (+ 13401us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.047999 (+ 4416us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28,"tablet-delete.queue_time_us":5439460}
I20250110 02:51:29.049188 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 27a7885a80414f2193400672aa16a911 (table default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]) successfully deleted
I20250110 02:51:29.046017 1184 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.049898 1184 ts_tablet_manager.cc:1397] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250110 02:51:29.043043 996 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.050166 1188 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.044601 2352 raft_consensus.cc:2238] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.050771 1184 raft_consensus.cc:357] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.050933 1184 raft_consensus.cc:383] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.050969 1184 raft_consensus.cc:738] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.051152 1184 consensus_queue.cc:260] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.051543 1184 ts_tablet_manager.cc:1428] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:29.051702 1188 tablet_bootstrap.cc:654] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.051723 1184 tablet_bootstrap.cc:492] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.052224 2735 raft_consensus.cc:2743] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.052763 1184 tablet_bootstrap.cc:654] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.055018 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 78db9d8dc1a04c7a98f74c81513d8fb8 (DEFAULT_TABLE table=default.Table_3018fa039953413a899fe3a942dcbb94 [id=44e00ca5f1c54c94ab6a95ad3b0541ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.054586 2357 raft_consensus.cc:2238] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.056414 2357 raft_consensus.cc:2267] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.056972 2357 ts_tablet_manager.cc:1905] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.067423 1188 tablet_bootstrap.cc:492] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.067693 1188 ts_tablet_manager.cc:1397] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20250110 02:51:29.068825 1188 raft_consensus.cc:357] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.069221 1188 raft_consensus.cc:383] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.069379 1188 raft_consensus.cc:738] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.069785 1188 consensus_queue.cc:260] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.071067 1188 ts_tablet_manager.cc:1428] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:29.071668 1188 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.073657 768 raft_consensus.cc:2463] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:29.076002 2224 tablet_replica.cc:331] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.076227 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78db9d8dc1a04c7a98f74c81513d8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.076295 2224 raft_consensus.cc:2238] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.077569 1188 tablet_bootstrap.cc:654] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.084596 2224 raft_consensus.cc:2267] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.085076 2224 ts_tablet_manager.cc:1905] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.087921 894 tablet_service.cc:1467] Processing CreateTablet for tablet d3e88c54b24946f4ac5ee735f888669a (DEFAULT_TABLE table=default.Table_6e05f69028df49f0afb60b28409b917a [id=2515a969af924528b4d6007b35b01555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.088354 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e88c54b24946f4ac5ee735f888669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.091743 2224 ts_tablet_manager.cc:1918] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:29.091948 2224 log.cc:1198] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:29.092283 2224 ts_tablet_manager.cc:1939] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.092892 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 156) took 5507 ms. Trace:
I20250110 02:51:29.093058 2224 rpcz_store.cc:276] 0110 02:51:23.585128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.586102 (+ 974us) service_pool.cc:225] Handling call
0110 02:51:29.049521 (+5463419us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:29.075950 (+ 26429us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.087881 (+ 11931us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.091659 (+ 3778us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.092839 (+ 1180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8165,"tablet-delete.queue_time_us":5463256}
I20250110 02:51:29.093650 2224 tablet_replica.cc:331] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.093777 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 8eb9ead3175140028e584e1ccb4874de (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]) successfully deleted
I20250110 02:51:29.093899 2224 raft_consensus.cc:2238] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.094221 2224 raft_consensus.cc:2267] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.094708 2224 ts_tablet_manager.cc:1905] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.095460 2456 consensus_queue.cc:1035] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.096030 2378 raft_consensus.cc:2743] T 6479d71a5142436b8c6ea2f76c221b33 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.096448 2352 raft_consensus.cc:2267] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.097191 2352 ts_tablet_manager.cc:1905] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.092283 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 7dec54ccb59846dfbcbcb46c624c4b23 (DEFAULT_TABLE table=default.Table_7927766540604688908ced9878eecd08 [id=33e6811488424fc589900d0d02729e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.098666 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec54ccb59846dfbcbcb46c624c4b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.100878 1181 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.104635 1181 ts_tablet_manager.cc:1397] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.119s user 0.019s sys 0.011s
I20250110 02:51:29.105279 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet bb595013e809480a814cb9c7aed856e6
I20250110 02:51:29.105297 1181 raft_consensus.cc:357] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.105551 1181 raft_consensus.cc:383] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.105644 1181 raft_consensus.cc:738] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.105845 1181 consensus_queue.cc:260] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.106803 1181 ts_tablet_manager.cc:1428] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:29.106923 1456 catalog_manager.cc:6596] ReplaceTablet: tablet bb595013e809480a814cb9c7aed856e6 deleted and replaced by tablet 9e28d8769f364e10aad54b60fb221f35
I20250110 02:51:29.107075 1181 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.107806 1456 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_e3e5bb662c644b5c8d10bd0f63ffe818" } new_table_name: "default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd" modify_external_catalogs: true
I20250110 02:51:29.108713 1181 tablet_bootstrap.cc:654] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.109706 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:29.110535 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_3fbaaf982d4c443a8d174437ef013ab2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:29.111181 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fbaaf982d4c443a8d174437ef013ab2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:29.112291 894 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:29.074911 1184 tablet_bootstrap.cc:492] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.117563 1184 ts_tablet_manager.cc:1397] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20250110 02:51:29.118252 1184 raft_consensus.cc:357] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.118438 1184 raft_consensus.cc:383] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.118484 1184 raft_consensus.cc:738] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.118654 1184 consensus_queue.cc:260] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.119117 1184 ts_tablet_manager.cc:1428] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:29.119364 1184 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.120193 2224 ts_tablet_manager.cc:1918] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.120270 2224 log.cc:1198] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:29.120573 1184 tablet_bootstrap.cc:654] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.120568 2224 ts_tablet_manager.cc:1939] T b2163c343a694f5ca04ac9f7610d25e3 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.120837 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb
I20250110 02:51:29.121122 2549 raft_consensus.cc:491] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.121225 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 150) took 5538 ms. Trace:
I20250110 02:51:29.121219 2549 raft_consensus.cc:513] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.121348 1450 catalog_manager.cc:6596] ReplaceTablet: tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb deleted and replaced by tablet e4d8527249404ac4be01021778e64fbe
I20250110 02:51:29.121289 2224 rpcz_store.cc:276] 0110 02:51:23.582465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.610854 (+ 28389us) service_pool.cc:225] Handling call
0110 02:51:29.093608 (+5482754us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:29.093618 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.097300 (+ 3682us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.120118 (+ 22818us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.121209 (+ 1091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5482563}
I20250110 02:51:29.121537 2224 tablet_replica.cc:331] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.121699 2224 raft_consensus.cc:2238] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.121755 2549 leader_election.cc:290] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.121990 2224 raft_consensus.cc:2267] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.122812 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b2163c343a694f5ca04ac9f7610d25e3 (table default.Table_868905ee96fe4989884cf84f8711954e [id=699a68566f294c94b7e18c3a16df6c9a]) successfully deleted
I20250110 02:51:29.123589 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8527249404ac4be01021778e64fbe) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:29.124264 1181 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.124358 1181 ts_tablet_manager.cc:1397] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250110 02:51:29.124892 1181 raft_consensus.cc:357] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.125098 1181 raft_consensus.cc:383] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.125186 1181 raft_consensus.cc:738] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.125393 1181 consensus_queue.cc:260] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.086655 742 tablet_service.cc:1467] Processing CreateTablet for tablet 78db9d8dc1a04c7a98f74c81513d8fb8 (DEFAULT_TABLE table=default.Table_3018fa039953413a899fe3a942dcbb94 [id=44e00ca5f1c54c94ab6a95ad3b0541ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.125900 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78db9d8dc1a04c7a98f74c81513d8fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.125964 1181 ts_tablet_manager.cc:1428] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:29.126163 1181 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.087702 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7dec54ccb59846dfbcbcb46c624c4b23 (DEFAULT_TABLE table=default.Table_7927766540604688908ced9878eecd08 [id=33e6811488424fc589900d0d02729e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.126672 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec54ccb59846dfbcbcb46c624c4b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.127508 1181 tablet_bootstrap.cc:654] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.128392 2224 ts_tablet_manager.cc:1905] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.106947 740 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:29.131214 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.131340 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8527249404ac4be01021778e64fbe) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:29.132071 709 leader_election.cc:343] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.133702 2224 ts_tablet_manager.cc:1918] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:29.133795 2224 log.cc:1198] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/5f229f546a8d4a09b93f8b46c893d14a
I20250110 02:51:29.134132 2224 ts_tablet_manager.cc:1939] T 5f229f546a8d4a09b93f8b46c893d14a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.134729 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.134943 1061 raft_consensus.cc:2463] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:29.136936 711 leader_election.cc:304] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.139189 2777 raft_consensus.cc:2798] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.139302 2777 raft_consensus.cc:491] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.139377 2777 raft_consensus.cc:3054] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.140612 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 161) took 5470 ms. Trace:
I20250110 02:51:29.140973 2224 rpcz_store.cc:276] 0110 02:51:23.670219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:23.670281 (+ 62us) service_pool.cc:225] Handling call
0110 02:51:29.121500 (+5451219us) ts_tablet_manager.cc:1145] Deleting tablet 5f229f546a8d4a09b93f8b46c893d14a
0110 02:51:29.121507 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.130982 (+ 9475us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.133640 (+ 2658us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.140596 (+ 6956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5451018}
I20250110 02:51:29.141734 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8527249404ac4be01021778e64fbe) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:29.141907 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 171) took 5006 ms. Trace:
I20250110 02:51:29.141932 2777 raft_consensus.cc:513] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.144671 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:29.144791 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:29.145186 2777 leader_election.cc:290] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.145215 1057 raft_consensus.cc:3054] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:29.145423 709 leader_election.cc:343] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.147955 1057 raft_consensus.cc:2463] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:29.148131 2224 rpcz_store.cc:276] 0110 02:51:24.135904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.135996 (+ 92us) service_pool.cc:225] Handling call
0110 02:51:29.141780 (+5005784us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:29.141792 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.141891 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5005630}
I20250110 02:51:29.148377 711 leader_election.cc:304] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
W20250110 02:51:29.148787 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 64153.2us
W20250110 02:51:29.148829 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because the tablet was not found. No further retry: Not found: Tablet not found: 0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:29.148829 2777 raft_consensus.cc:2798] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 02:51:29.148993 996 leader_election.cc:343] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:29.149201 996 leader_election.cc:304] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.149237 2777 raft_consensus.cc:695] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
W20250110 02:51:29.149410 996 raft_consensus.cc:2681] T 8eb9ead3175140028e584e1ccb4874de P 8899dd1aeb384e4ea4523f5fd225c888: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 02:51:29.149261 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 169) took 5013 ms. Trace:
I20250110 02:51:29.149581 2777 consensus_queue.cc:237] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.149668 2224 rpcz_store.cc:276] 0110 02:51:24.135712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.136202 (+ 490us) service_pool.cc:225] Handling call
0110 02:51:29.149130 (+5012928us) ts_tablet_manager.cc:1145] Deleting tablet e015a41a03f84c91a45e46e4b7b5e7a9
0110 02:51:29.149139 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.149244 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5012755}
W20250110 02:51:29.150043 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e015a41a03f84c91a45e46e4b7b5e7a9 already in progress: creating tablet
I20250110 02:51:29.150534 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 173) took 4997 ms. Trace:
I20250110 02:51:29.150888 2224 rpcz_store.cc:276] 0110 02:51:24.152735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.153094 (+ 359us) service_pool.cc:225] Handling call
0110 02:51:29.150455 (+4997361us) ts_tablet_manager.cc:1145] Deleting tablet 46a4745356e3423f9d4892d242da1c48
0110 02:51:29.150463 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.150520 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4997177}
W20250110 02:51:29.151566 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 46a4745356e3423f9d4892d242da1c48 because the tablet was not found. No further retry: Not found: Tablet not found: 46a4745356e3423f9d4892d242da1c48
I20250110 02:51:29.151937 2224 tablet_replica.cc:331] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.152354 2224 raft_consensus.cc:2238] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.152745 2224 raft_consensus.cc:2267] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.153525 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d3e88c54b24946f4ac5ee735f888669a (DEFAULT_TABLE table=default.Table_6e05f69028df49f0afb60b28409b917a [id=2515a969af924528b4d6007b35b01555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.153872 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e88c54b24946f4ac5ee735f888669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.153918 742 tablet_service.cc:1467] Processing CreateTablet for tablet d3e88c54b24946f4ac5ee735f888669a (DEFAULT_TABLE table=default.Table_6e05f69028df49f0afb60b28409b917a [id=2515a969af924528b4d6007b35b01555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.154215 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e88c54b24946f4ac5ee735f888669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.154881 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2985e166534e68a073fa18c0ce4b6c (DEFAULT_TABLE table=default.Table_51799375e04649b5955644f75e31c140 [id=97b5afac99f94b57977a19cd667d186e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.155172 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2985e166534e68a073fa18c0ce4b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.158694 894 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2985e166534e68a073fa18c0ce4b6c (DEFAULT_TABLE table=default.Table_51799375e04649b5955644f75e31c140 [id=97b5afac99f94b57977a19cd667d186e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.159185 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2985e166534e68a073fa18c0ce4b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.159969 2224 ts_tablet_manager.cc:1905] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.163300 2378 raft_consensus.cc:491] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.163444 2378 raft_consensus.cc:513] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.164351 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65285bce60fd4015b085deb7442eea6e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.164518 1057 raft_consensus.cc:2463] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
W20250110 02:51:29.165277 853 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:29.165490 853 leader_election.cc:304] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.165711 2691 raft_consensus.cc:2798] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.165773 2691 raft_consensus.cc:491] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.165841 2691 raft_consensus.cc:3054] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.168239 2691 raft_consensus.cc:513] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.168833 2691 leader_election.cc:290] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:29.170248 853 leader_election.cc:343] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:29.171952 2456 raft_consensus.cc:491] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.172060 2456 raft_consensus.cc:513] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.172796 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.173206 994 leader_election.cc:343] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.173244 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65285bce60fd4015b085deb7442eea6e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:29.173442 1057 raft_consensus.cc:3054] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.173471 2456 leader_election.cc:290] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.173907 2378 leader_election.cc:290] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.176095 1057 raft_consensus.cc:2463] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:29.176434 853 leader_election.cc:304] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.176859 2691 raft_consensus.cc:2798] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.176968 2691 raft_consensus.cc:695] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:29.177178 2691 consensus_queue.cc:237] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.180842 756 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2985e166534e68a073fa18c0ce4b6c (DEFAULT_TABLE table=default.Table_51799375e04649b5955644f75e31c140 [id=97b5afac99f94b57977a19cd667d186e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.181427 889 tablet_service.cc:1467] Processing CreateTablet for tablet 30645a0642bc43ff957368f928b877d1 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.181668 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2985e166534e68a073fa18c0ce4b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.181782 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30645a0642bc43ff957368f928b877d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.181964 740 tablet_service.cc:1467] Processing CreateTablet for tablet 30645a0642bc43ff957368f928b877d1 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.182731 883 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53b838be846c7b3ac44dc75aa3730 (DEFAULT_TABLE table=default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.182957 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30645a0642bc43ff957368f928b877d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.183048 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e53b838be846c7b3ac44dc75aa3730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.186600 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53b838be846c7b3ac44dc75aa3730 (DEFAULT_TABLE table=default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.186600 748 tablet_service.cc:1467] Processing CreateTablet for tablet 70e53b838be846c7b3ac44dc75aa3730 (DEFAULT_TABLE table=default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.186925 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e53b838be846c7b3ac44dc75aa3730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.186935 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e53b838be846c7b3ac44dc75aa3730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.188225 2549 raft_consensus.cc:491] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.188349 2549 raft_consensus.cc:513] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.188982 2549 leader_election.cc:290] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.190956 753 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (DEFAULT_TABLE table=default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.191251 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.192952 2735 consensus_queue.cc:1035] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.195853 2600 consensus_queue.cc:1035] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.195976 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.196441 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (DEFAULT_TABLE table=default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.196761 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.196934 2345 raft_consensus.cc:491] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.196997 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (DEFAULT_TABLE table=default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.197043 2345 raft_consensus.cc:513] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.197187 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3c25ee678e194a0ca1e2291512cbc478 (DEFAULT_TABLE table=default.Table_428c37996d7b406083d3f703508d1729 [id=50d1bd19074a4c4b973a476f0343f67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.197557 2345 leader_election.cc:290] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.197932 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.201604 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:29.201925 709 leader_election.cc:343] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:29.202636 711 leader_election.cc:343] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.202718 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3e52914a424cb3a8bf422b3a56fb22" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.202781 2224 ts_tablet_manager.cc:1918] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:29.202955 770 raft_consensus.cc:2388] T 9d3e52914a424cb3a8bf422b3a56fb22 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:29.203195 2224 log.cc:1198] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:29.204090 853 leader_election.cc:304] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.204715 2773 raft_consensus.cc:3054] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.204852 2224 ts_tablet_manager.cc:1939] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.207218 2773 raft_consensus.cc:2743] T 9d3e52914a424cb3a8bf422b3a56fb22 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.208528 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 172) took 5055 ms. Trace:
I20250110 02:51:29.209045 2224 rpcz_store.cc:276] 0110 02:51:24.152618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.153546 (+ 928us) service_pool.cc:225] Handling call
0110 02:51:29.151890 (+4998344us) ts_tablet_manager.cc:1145] Deleting tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
0110 02:51:29.151900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.179585 (+ 27685us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.182501 (+ 2916us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.208509 (+ 26008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4998184}
I20250110 02:51:29.210860 2224 tablet_replica.cc:331] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.202762 711 leader_election.cc:304] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.211704 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (table default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]) successfully deleted
I20250110 02:51:29.212405 2549 raft_consensus.cc:2743] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.212190 2224 raft_consensus.cc:2238] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.213155 895 tablet_service.cc:1467] Processing CreateTablet for tablet 7dec54ccb59846dfbcbcb46c624c4b23 (DEFAULT_TABLE table=default.Table_7927766540604688908ced9878eecd08 [id=33e6811488424fc589900d0d02729e71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.213021 2224 raft_consensus.cc:2267] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.214296 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c25ee678e194a0ca1e2291512cbc478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.214550 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.221874 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dec54ccb59846dfbcbcb46c624c4b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.224606 2773 raft_consensus.cc:491] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.224727 2773 raft_consensus.cc:513] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.225276 2773 leader_election.cc:290] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.226722 2224 ts_tablet_manager.cc:1905] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:29.226769 853 leader_election.cc:343] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
W20250110 02:51:29.226958 853 leader_election.cc:343] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:29.227048 853 leader_election.cc:304] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.227444 2773 raft_consensus.cc:2743] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.227562 2345 raft_consensus.cc:491] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.227686 2345 raft_consensus.cc:513] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.228185 2345 leader_election.cc:290] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:29.229972 853 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 240b64d551a548e884ee2a7238c0363a
W20250110 02:51:29.230216 853 leader_election.cc:343] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 240b64d551a548e884ee2a7238c0363a
I20250110 02:51:29.230333 853 leader_election.cc:304] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.230698 2345 raft_consensus.cc:2743] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:29.241336 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.241650 768 raft_consensus.cc:2463] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:29.241835 2224 ts_tablet_manager.cc:1918] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:29.243042 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_4f07307b0947487b9111c6249489de28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:29.243672 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f07307b0947487b9111c6249489de28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:29.246024 1184 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.246227 1184 ts_tablet_manager.cc:1397] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.000s
I20250110 02:51:29.246579 2224 log.cc:1198] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:29.246902 2224 ts_tablet_manager.cc:1939] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.247109 1184 raft_consensus.cc:357] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.247339 1184 raft_consensus.cc:383] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.247382 1184 raft_consensus.cc:738] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.247592 1184 consensus_queue.cc:260] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.248142 1184 ts_tablet_manager.cc:1428] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:29.248864 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 170) took 5113 ms. Trace:
I20250110 02:51:29.115646 2357 ts_tablet_manager.cc:1918] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.249502 2357 log.cc:1198] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/9507a5283d8f4d07a293ba474c642298
I20250110 02:51:29.249861 2357 ts_tablet_manager.cc:1939] T 9507a5283d8f4d07a293ba474c642298 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.249985 2224 rpcz_store.cc:276] 0110 02:51:24.135818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.135969 (+ 151us) service_pool.cc:225] Handling call
0110 02:51:29.210494 (+5074525us) ts_tablet_manager.cc:1145] Deleting tablet 6e0b2ec1d5814144bc58261e6c6e8d35
0110 02:51:29.210738 (+ 244us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.238878 (+ 28140us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.241749 (+ 2871us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.248840 (+ 7091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5035853}
I20250110 02:51:29.250852 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (table default.Table_c2f177ce9d6a420bbc77b33929ccaa7a [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]) successfully deleted
I20250110 02:51:29.250917 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:29.253780 2224 tablet_replica.cc:331] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.254952 2224 raft_consensus.cc:2238] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.255003 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 30645a0642bc43ff957368f928b877d1 (DEFAULT_TABLE table=default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.255214 2224 raft_consensus.cc:2267] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.255324 2478 raft_consensus.cc:491] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.254691 1181 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.255509 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30645a0642bc43ff957368f928b877d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.255568 1181 ts_tablet_manager.cc:1397] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.012s
I20250110 02:51:29.255764 2478 raft_consensus.cc:513] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.256338 1181 raft_consensus.cc:357] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.256525 1181 raft_consensus.cc:383] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.256635 1181 raft_consensus.cc:738] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.256814 1181 consensus_queue.cc:260] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.256909 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00e97e496b54dcf82b7182591d6f4aa" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.257336 1181 ts_tablet_manager.cc:1428] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250110 02:51:29.257444 994 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.257712 1181 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.257761 2478 leader_election.cc:290] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.259440 1181 tablet_bootstrap.cc:654] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.259896 2224 ts_tablet_manager.cc:1905] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.262246 2352 ts_tablet_manager.cc:1918] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.265594 1440 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c" } modify_external_catalogs: true
I20250110 02:51:29.265851 1440 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c" } modify_external_catalogs: true
I20250110 02:51:29.272503 1184 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.274346 1184 tablet_bootstrap.cc:654] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.274955 996 leader_election.cc:304] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.275261 2456 raft_consensus.cc:2798] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.275367 2456 raft_consensus.cc:695] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:29.275585 2456 consensus_queue.cc:237] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.276155 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 130) took 5022 ms. Trace:
I20250110 02:51:29.276360 2357 rpcz_store.cc:276] 0110 02:51:24.253928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.254993 (+ 1065us) service_pool.cc:225] Handling call
0110 02:51:29.045337 (+4790344us) ts_tablet_manager.cc:1145] Deleting tablet 9507a5283d8f4d07a293ba474c642298
0110 02:51:29.045347 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.066116 (+ 20769us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.115557 (+ 49441us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.276140 (+160583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4790238}
I20250110 02:51:29.276933 2357 tablet_replica.cc:331] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.276885 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 9507a5283d8f4d07a293ba474c642298 (table default.Table_c36e12f8b3e64b41980128408d36846f [id=f7ac303fffd946368166a0a04e8056fa]) successfully deleted
I20250110 02:51:29.277190 2357 raft_consensus.cc:2238] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.277545 2357 raft_consensus.cc:2267] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.278133 2357 ts_tablet_manager.cc:1905] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.283622 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3c25ee678e194a0ca1e2291512cbc478 (DEFAULT_TABLE table=default.Table_428c37996d7b406083d3f703508d1729 [id=50d1bd19074a4c4b973a476f0343f67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.284050 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c25ee678e194a0ca1e2291512cbc478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:29.286362 2600 raft_consensus.cc:491] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.286505 2600 raft_consensus.cc:513] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.287088 2600 leader_election.cc:290] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.289024 2224 ts_tablet_manager.cc:1918] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.289373 2224 log.cc:1198] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/19e4af23720048fe98b3231ba402d45c
I20250110 02:51:29.289817 2224 ts_tablet_manager.cc:1939] T 19e4af23720048fe98b3231ba402d45c P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.290155 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 176) took 5115 ms. Trace:
I20250110 02:51:29.290961 2224 rpcz_store.cc:276] 0110 02:51:24.174839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.174952 (+ 113us) service_pool.cc:225] Handling call
0110 02:51:29.253732 (+5078780us) ts_tablet_manager.cc:1145] Deleting tablet 19e4af23720048fe98b3231ba402d45c
0110 02:51:29.253742 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.269537 (+ 15795us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.288951 (+ 19414us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.290101 (+ 1150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5078643}
I20250110 02:51:29.291358 2224 tablet_replica.cc:331] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.290897 1181 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.291472 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 19e4af23720048fe98b3231ba402d45c (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
I20250110 02:51:29.291553 1181 ts_tablet_manager.cc:1397] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.004s
I20250110 02:51:29.292239 1181 raft_consensus.cc:357] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.292430 1181 raft_consensus.cc:383] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.292493 1181 raft_consensus.cc:738] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.292692 1181 consensus_queue.cc:260] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.293207 1181 ts_tablet_manager.cc:1428] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:29.293442 1181 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.293800 2777 raft_consensus.cc:491] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.293984 2777 raft_consensus.cc:513] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.295063 2224 raft_consensus.cc:2238] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.295207 2777 leader_election.cc:290] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.295854 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.296100 2224 raft_consensus.cc:2267] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.295925 1181 tablet_bootstrap.cc:654] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:29.296396 711 leader_election.cc:343] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.299175 2224 ts_tablet_manager.cc:1905] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.302917 756 tablet_service.cc:1467] Processing CreateTablet for tablet 3c25ee678e194a0ca1e2291512cbc478 (DEFAULT_TABLE table=default.Table_428c37996d7b406083d3f703508d1729 [id=50d1bd19074a4c4b973a476f0343f67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:29.303701 2600 consensus_queue.cc:1035] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.310966 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c25ee678e194a0ca1e2291512cbc478. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:29.322458 994 consensus_peers.cc:487] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:29.322960 996 leader_election.cc:304] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.326944 2478 raft_consensus.cc:2798] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.327147 2478 raft_consensus.cc:491] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.327381 2478 raft_consensus.cc:3054] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.331379 2478 raft_consensus.cc:513] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.338275 2224 ts_tablet_manager.cc:1918] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.343807 1184 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.344003 1184 ts_tablet_manager.cc:1397] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20250110 02:51:29.344730 1184 raft_consensus.cc:357] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.344967 1184 raft_consensus.cc:383] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.345037 1184 raft_consensus.cc:738] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.345688 1184 consensus_queue.cc:260] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.347679 1184 ts_tablet_manager.cc:1428] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:29.348075 1184 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.355604 1184 tablet_bootstrap.cc:654] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.355844 2224 log.cc:1198] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:29.356395 2224 ts_tablet_manager.cc:1939] T b382d634db9642df8e00c5ccaa5787a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.357277 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 177) took 5182 ms. Trace:
I20250110 02:51:29.357445 2224 rpcz_store.cc:276] 0110 02:51:24.174911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.175111 (+ 200us) service_pool.cc:225] Handling call
0110 02:51:29.291283 (+5116172us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:29.291292 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.301900 (+ 10608us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.338086 (+ 36186us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.357239 (+ 19153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5116074}
I20250110 02:51:29.357919 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 174) took 5205 ms. Trace:
I20250110 02:51:29.357928 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet b382d634db9642df8e00c5ccaa5787a9 (table default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]) successfully deleted
I20250110 02:51:29.359830 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.361146 768 raft_consensus.cc:2463] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:29.361004 2357 ts_tablet_manager.cc:1918] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.361524 2357 log.cc:1198] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:29.362983 2357 ts_tablet_manager.cc:1939] T 1390f867a3a8436092d1920daff4c5b0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.363766 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 135) took 5109 ms. Trace:
I20250110 02:51:29.363909 2357 rpcz_store.cc:276] 0110 02:51:24.254442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.255128 (+ 686us) service_pool.cc:225] Handling call
0110 02:51:29.276889 (+5021761us) ts_tablet_manager.cc:1145] Deleting tablet 1390f867a3a8436092d1920daff4c5b0
0110 02:51:29.276899 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.281916 (+ 5017us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.360900 (+ 78984us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.363751 (+ 2851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5021633}
I20250110 02:51:29.364367 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1390f867a3a8436092d1920daff4c5b0 (table default.Table_f10221708fa5417d8300ada3f8eb0817 [id=81d7d3112de44cae840ee0f1f86e6f2f]) successfully deleted
I20250110 02:51:29.350358 2738 consensus_queue.cc:1035] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.364760 2224 rpcz_store.cc:276] 0110 02:51:24.152816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.153720 (+ 904us) service_pool.cc:225] Handling call
0110 02:51:29.357779 (+5204059us) ts_tablet_manager.cc:1145] Deleting tablet d6fd370138ac4039be55642f1b4c5607
0110 02:51:29.357787 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.357878 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5162476}
I20250110 02:51:29.365106 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
W20250110 02:51:29.365272 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because the tablet was not found. No further retry: Not found: Tablet not found: d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:29.367349 2357 tablet_replica.cc:331] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.367529 2224 tablet_replica.cc:331] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.368073 2357 raft_consensus.cc:2238] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.368250 2224 raft_consensus.cc:2238] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.368525 2357 raft_consensus.cc:2267] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.368793 2224 raft_consensus.cc:2267] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.369694 2357 ts_tablet_manager.cc:1905] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.370522 2224 ts_tablet_manager.cc:1905] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.338478 2478 leader_election.cc:290] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.378782 1184 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.379024 1184 ts_tablet_manager.cc:1397] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.001s
I20250110 02:51:29.384341 1184 raft_consensus.cc:357] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.384660 1184 raft_consensus.cc:383] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.384768 1184 raft_consensus.cc:738] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.385087 1184 consensus_queue.cc:260] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.386662 1184 ts_tablet_manager.cc:1428] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250110 02:51:29.386968 1184 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.287707 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.295050 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.303498 1188 tablet_bootstrap.cc:492] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.387401 1188 ts_tablet_manager.cc:1397] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.316s user 0.030s sys 0.031s
W20250110 02:51:29.387766 709 leader_election.cc:343] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.387903 709 leader_election.cc:304] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.388088 1188 raft_consensus.cc:357] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:29.388192 994 leader_election.cc:343] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.388281 1188 raft_consensus.cc:383] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.388329 1188 raft_consensus.cc:738] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.388520 2777 raft_consensus.cc:2743] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.388505 1188 consensus_queue.cc:260] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.388993 1188 ts_tablet_manager.cc:1428] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:29.389225 1188 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.389417 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:29.389580 774 raft_consensus.cc:3054] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.389585 2352 log.cc:1198] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:29.313668 2378 raft_consensus.cc:491] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.318435 2345 raft_consensus.cc:491] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.332604 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:29.390475 2345 raft_consensus.cc:513] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:29.390810 994 leader_election.cc:343] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.391047 2345 leader_election.cc:290] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.390477 2378 raft_consensus.cc:513] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.391395 1188 tablet_bootstrap.cc:654] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.391690 2378 leader_election.cc:290] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.392328 774 raft_consensus.cc:2463] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:29.392683 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.392992 1061 raft_consensus.cc:2388] T c1608d90ceb64602889b5705ba4ab55d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:29.392709 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.394338 2549 raft_consensus.cc:491] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.394482 2549 raft_consensus.cc:513] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.395131 2549 leader_election.cc:290] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.395761 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12ce7a0dc4f4f3586a2f918d31234e1" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.395783 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7ec8c7891b4a2c9fcb3cc2eeedd29a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.395946 764 raft_consensus.cc:2388] T b12ce7a0dc4f4f3586a2f918d31234e1 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
W20250110 02:51:29.396245 709 leader_election.cc:343] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.398634 853 leader_election.cc:304] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.399102 1188 tablet_bootstrap.cc:492] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.399183 2378 raft_consensus.cc:3054] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.399230 1188 ts_tablet_manager.cc:1397] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250110 02:51:29.399926 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7ec8c7891b4a2c9fcb3cc2eeedd29a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.400039 1188 raft_consensus.cc:357] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.400199 1188 raft_consensus.cc:383] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.400249 1188 raft_consensus.cc:738] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:29.400431 711 leader_election.cc:343] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.400512 711 leader_election.cc:304] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.400410 1188 consensus_queue.cc:260] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.400885 1188 ts_tablet_manager.cc:1428] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:29.400871 2777 raft_consensus.cc:2743] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.401103 1188 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.401126 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1608d90ceb64602889b5705ba4ab55d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.401417 2378 raft_consensus.cc:2743] T b12ce7a0dc4f4f3586a2f918d31234e1 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.402238 853 leader_election.cc:304] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.402487 1188 tablet_bootstrap.cc:654] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.402760 2345 raft_consensus.cc:3054] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.405040 2345 raft_consensus.cc:2743] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.405843 1184 tablet_bootstrap.cc:654] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.412168 2378 raft_consensus.cc:491] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:29.412312 2378 raft_consensus.cc:513] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.412534 2357 ts_tablet_manager.cc:1918] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:29.412685 2357 log.cc:1198] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f63d69d04b494a848653636c9e259105
I20250110 02:51:29.412933 2378 leader_election.cc:290] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.413173 2357 ts_tablet_manager.cc:1939] T f63d69d04b494a848653636c9e259105 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:29.413959 853 leader_election.cc:343] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 95b14d7592024a34baf7756d92e546d9
W20250110 02:51:29.414139 853 leader_election.cc:343] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 95b14d7592024a34baf7756d92e546d9
I20250110 02:51:29.414242 853 leader_election.cc:304] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.414682 2378 raft_consensus.cc:2743] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:29.417804 2224 ts_tablet_manager.cc:1918] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.417994 2224 log.cc:1198] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:29.413527 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 129) took 5159 ms. Trace:
I20250110 02:51:29.418737 2357 rpcz_store.cc:276] 0110 02:51:24.253842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.255456 (+ 1614us) service_pool.cc:225] Handling call
0110 02:51:29.367309 (+5111853us) ts_tablet_manager.cc:1145] Deleting tablet f63d69d04b494a848653636c9e259105
0110 02:51:29.367318 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.373334 (+ 6016us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.412363 (+ 39029us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.413513 (+ 1150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111681}
I20250110 02:51:29.418954 1188 tablet_bootstrap.cc:492] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.419221 2224 ts_tablet_manager.cc:1939] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.419955 2357 tablet_replica.cc:331] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.419696 1188 ts_tablet_manager.cc:1397] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20250110 02:51:29.420756 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 180) took 5103 ms. Trace:
I20250110 02:51:29.421483 2357 raft_consensus.cc:2238] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.422240 2357 raft_consensus.cc:2267] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.422736 2357 ts_tablet_manager.cc:1905] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.422878 2224 rpcz_store.cc:276] 0110 02:51:24.317713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.317789 (+ 76us) service_pool.cc:225] Handling call
0110 02:51:29.367453 (+5049664us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:29.367463 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.381267 (+ 13804us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.417692 (+ 36425us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.420707 (+ 3015us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5047069}
I20250110 02:51:29.426414 1188 raft_consensus.cc:357] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.427404 1188 raft_consensus.cc:383] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.427810 1188 raft_consensus.cc:738] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.426349 2738 consensus_queue.cc:1035] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.428620 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:29.438747 2224 tablet_replica.cc:331] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.422087 1184 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.439190 1184 ts_tablet_manager.cc:1397] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20250110 02:51:29.439479 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (table default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]) successfully deleted
I20250110 02:51:29.440342 1184 raft_consensus.cc:357] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.440670 1184 raft_consensus.cc:383] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.440770 1184 raft_consensus.cc:738] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.439299 1188 consensus_queue.cc:260] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.441053 1184 consensus_queue.cc:260] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.441644 1184 ts_tablet_manager.cc:1428] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:29.441926 1184 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.444178 1184 tablet_bootstrap.cc:654] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.444000 1188 ts_tablet_manager.cc:1428] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.024s user 0.001s sys 0.002s
I20250110 02:51:29.445042 2224 raft_consensus.cc:2238] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.446064 1188 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.447548 2224 raft_consensus.cc:2267] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.449779 2224 ts_tablet_manager.cc:1905] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.451898 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_becbebb399f04a568ef57354161f2bb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:29.452539 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_becbebb399f04a568ef57354161f2bb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:29.463001 2199 raft_consensus.cc:491] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.463196 2199 raft_consensus.cc:513] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.463944 2199 leader_election.cc:290] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.464679 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.464694 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.465147 711 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:29.465260 709 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.465354 709 leader_election.cc:304] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.465730 2199 raft_consensus.cc:2743] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.466605 1450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_bb82c27076c5492e9788b3e1b4a66c1a" } modify_external_catalogs: true
I20250110 02:51:29.467036 1188 tablet_bootstrap.cc:654] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.473758 2352 ts_tablet_manager.cc:1939] T cf711522866d46439d7ee69aa63d7f39 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:29.474095 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 164) took 5205 ms. Trace:
I20250110 02:51:29.474233 2352 rpcz_store.cc:276] 0110 02:51:24.268747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270680 (+ 1933us) service_pool.cc:225] Handling call
0110 02:51:28.983521 (+4712841us) ts_tablet_manager.cc:1145] Deleting tablet cf711522866d46439d7ee69aa63d7f39
0110 02:51:28.983530 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.258853 (+275323us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.262155 (+ 3302us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.474081 (+211926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4712735}
I20250110 02:51:29.475327 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 153) took 5207 ms. Trace:
I20250110 02:51:29.475435 2352 rpcz_store.cc:276] 0110 02:51:24.267775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270805 (+ 3030us) service_pool.cc:225] Handling call
0110 02:51:29.474982 (+5204177us) ts_tablet_manager.cc:1145] Deleting tablet dba0cb7970f1453dac256b255ed7cac0
0110 02:51:29.474991 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.475311 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5204065}
I20250110 02:51:29.476318 2352 tablet_replica.cc:331] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:29.476852 2352 raft_consensus.cc:2238] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.477030 2352 raft_consensus.cc:2267] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.477684 2352 ts_tablet_manager.cc:1905] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.479559 2357 ts_tablet_manager.cc:1918] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.480023 2357 log.cc:1198] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:29.480434 2357 ts_tablet_manager.cc:1939] T bcc9a4a842914a6f878d01194cd9a5f7 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.481063 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 136) took 5225 ms. Trace:
I20250110 02:51:29.483304 2224 ts_tablet_manager.cc:1918] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.483484 2357 rpcz_store.cc:276] 0110 02:51:24.255415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.255662 (+ 247us) service_pool.cc:225] Handling call
0110 02:51:29.419848 (+5164186us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:29.419875 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.453721 (+ 33846us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.479465 (+ 25744us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.481049 (+ 1584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5164043}
I20250110 02:51:29.483428 2738 consensus_queue.cc:1035] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.483819 2224 log.cc:1198] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/9289de0630804f9aa60fdaf5f3f15f68
W20250110 02:51:29.484000 996 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:29.484143 996 leader_election.cc:304] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.484362 2738 raft_consensus.cc:2743] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.484723 996 leader_election.cc:304] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.485006 2738 raft_consensus.cc:2798] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.485155 2738 raft_consensus.cc:491] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.485287 2738 raft_consensus.cc:3054] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.485342 1188 tablet_bootstrap.cc:492] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.485908 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 128) took 5232 ms. Trace:
I20250110 02:51:29.485934 1188 ts_tablet_manager.cc:1397] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.002s
I20250110 02:51:29.488008 2738 raft_consensus.cc:513] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.488440 2224 ts_tablet_manager.cc:1939] T 9289de0630804f9aa60fdaf5f3f15f68 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.488574 2357 rpcz_store.cc:276] 0110 02:51:24.253710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.255923 (+ 2213us) service_pool.cc:225] Handling call
0110 02:51:29.485795 (+5229872us) ts_tablet_manager.cc:1145] Deleting tablet a0ddab9e847445999ec97eac4352d9c2
0110 02:51:29.485804 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.485893 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5229736}
I20250110 02:51:29.488965 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:29.489065 2738 leader_election.cc:290] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:29.489405 994 leader_election.cc:343] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.489717 2357 tablet_replica.cc:331] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.491269 2357 raft_consensus.cc:2238] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 2 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.491127 1188 raft_consensus.cc:357] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.492349 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 178) took 5174 ms. Trace:
I20250110 02:51:29.492584 2357 raft_consensus.cc:2267] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.492156 1188 raft_consensus.cc:383] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.492666 1188 raft_consensus.cc:738] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.493273 2224 rpcz_store.cc:276] 0110 02:51:24.317542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.318338 (+ 796us) service_pool.cc:225] Handling call
0110 02:51:29.438606 (+5120268us) ts_tablet_manager.cc:1145] Deleting tablet 9289de0630804f9aa60fdaf5f3f15f68
0110 02:51:29.438620 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.467518 (+ 28898us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.470293 (+ 2775us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.492299 (+ 22006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5119015}
I20250110 02:51:29.494149 2357 ts_tablet_manager.cc:1905] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.494498 1188 consensus_queue.cc:260] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.496315 2224 tablet_replica.cc:331] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.496912 2224 raft_consensus.cc:2238] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.499316 1188 ts_tablet_manager.cc:1428] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250110 02:51:29.500375 2224 raft_consensus.cc:2267] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.501210 2224 ts_tablet_manager.cc:1905] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.501749 1188 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.475121 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet cf711522866d46439d7ee69aa63d7f39 (table default.Table_08d86300a7254466a9bff50b3f333765 [id=b0585ee264c44482941397f09c2eebb9]) successfully deleted
I20250110 02:51:29.485617 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet bcc9a4a842914a6f878d01194cd9a5f7 (table default.Table_cea75045b3d94a008b85e2f3bbefdebc [id=aaf697b2152748c99b06cea713d17023]) successfully deleted
W20250110 02:51:29.502950 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba0cb7970f1453dac256b255ed7cac0 already in progress: creating tablet
I20250110 02:51:29.502923 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 9289de0630804f9aa60fdaf5f3f15f68 (table default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]) successfully deleted
W20250110 02:51:29.503348 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:29.503247 1450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_bb82c27076c5492e9788b3e1b4a66c1a" } modify_external_catalogs: true
I20250110 02:51:29.512478 2456 consensus_queue.cc:1035] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.514508 1188 tablet_bootstrap.cc:654] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.528091 2199 raft_consensus.cc:491] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.528254 2199 raft_consensus.cc:513] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.528836 2199 leader_election.cc:290] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.529458 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.530009 709 leader_election.cc:343] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.530858 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:29.531349 711 leader_election.cc:343] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.531447 711 leader_election.cc:304] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.531697 2199 raft_consensus.cc:2743] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.538651 2738 raft_consensus.cc:491] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.538946 2738 raft_consensus.cc:513] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.540014 2738 leader_election.cc:290] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.540021 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.540566 994 leader_election.cc:343] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:29.551386 1429 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet dc6b2654f86e4686b8a3034de02d11bf (table default.Table_99ac393aee99471a8a4a750750f72cc8 [id=08f5930b70fd4aeb869e1793dd946f43], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet dc6b2654f86e4686b8a3034de02d11bf consensus state has no leader
I20250110 02:51:29.552560 996 leader_election.cc:304] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.552878 2456 raft_consensus.cc:2798] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.553009 2456 raft_consensus.cc:695] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:29.553267 2456 consensus_queue.cc:237] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:29.558214 853 consensus_peers.cc:487] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:29.559444 908 raft_consensus.cc:3054] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.572039 1184 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.572340 1184 ts_tablet_manager.cc:1397] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.130s user 0.000s sys 0.012s
I20250110 02:51:29.576437 1184 raft_consensus.cc:357] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.576701 1181 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.576752 1184 raft_consensus.cc:383] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.576879 1181 ts_tablet_manager.cc:1397] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.284s user 0.060s sys 0.063s
I20250110 02:51:29.576903 1184 raft_consensus.cc:738] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.577122 2345 raft_consensus.cc:491] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:29.577291 2345 raft_consensus.cc:513] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.577672 2199 raft_consensus.cc:491] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.577626 1181 raft_consensus.cc:357] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.577793 1181 raft_consensus.cc:383] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.577837 1181 raft_consensus.cc:738] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.577769 2199 raft_consensus.cc:513] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.577981 2345 leader_election.cc:290] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.578006 1181 consensus_queue.cc:260] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.578289 2600 raft_consensus.cc:491] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.578433 2199 leader_election.cc:290] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.578387 2600 raft_consensus.cc:513] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.578955 2600 leader_election.cc:290] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:29.579459 1181 ts_tablet_manager.cc:1428] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:29.579694 853 leader_election.cc:343] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:29.579748 1181 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.577207 1184 consensus_queue.cc:260] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.580149 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.580575 1184 ts_tablet_manager.cc:1428] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250110 02:51:29.580544 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.580703 1057 raft_consensus.cc:2463] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:29.581084 1181 tablet_bootstrap.cc:654] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.581094 711 leader_election.cc:304] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:29.581326 2549 raft_consensus.cc:2798] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.581387 2549 raft_consensus.cc:491] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.581450 2549 raft_consensus.cc:3054] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:29.582621 994 leader_election.cc:343] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.580818 1184 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:29.583988 853 leader_election.cc:343] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:29.584095 853 leader_election.cc:304] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.585207 1184 tablet_bootstrap.cc:654] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.592144 2549 raft_consensus.cc:513] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.592931 2549 leader_election.cc:290] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.598038 2600 consensus_queue.cc:1035] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.598490 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:29.599054 709 leader_election.cc:343] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.600168 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:29.600335 1057 raft_consensus.cc:3054] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.601044 2549 consensus_queue.cc:1035] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.604790 1181 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.605526 1181 ts_tablet_manager.cc:1397] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20250110 02:51:29.606652 2224 ts_tablet_manager.cc:1918] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.606801 2224 log.cc:1198] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/302af84fd8374ffa96a7503b54849536
I20250110 02:51:29.607199 2224 ts_tablet_manager.cc:1939] T 302af84fd8374ffa96a7503b54849536 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.606662 1181 raft_consensus.cc:357] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.607987 1181 raft_consensus.cc:383] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.608171 1181 raft_consensus.cc:738] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.608570 1181 consensus_queue.cc:260] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.609658 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:29.609768 1181 ts_tablet_manager.cc:1428] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:29.609824 764 raft_consensus.cc:3054] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.609907 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.610008 1181 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:29.610400 709 leader_election.cc:343] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.611397 1181 tablet_bootstrap.cc:654] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.618808 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.619006 774 raft_consensus.cc:2463] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:29.621037 1181 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.621491 1181 ts_tablet_manager.cc:1397] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250110 02:51:29.621560 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:29.622292 1181 raft_consensus.cc:357] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.623298 1181 raft_consensus.cc:383] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.623477 1181 raft_consensus.cc:738] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.624102 1181 consensus_queue.cc:260] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.625051 1181 ts_tablet_manager.cc:1428] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:29.630735 1181 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.631822 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.632052 774 raft_consensus.cc:2388] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8899dd1aeb384e4ea4523f5fd225c888 in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:29.632694 1181 tablet_bootstrap.cc:654] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.633703 2199 consensus_queue.cc:1035] T c1608d90ceb64602889b5705ba4ab55d P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.640499 1181 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.641036 1181 ts_tablet_manager.cc:1397] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250110 02:51:29.641867 1181 raft_consensus.cc:357] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.645946 2345 raft_consensus.cc:2743] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:29.646426 1181 raft_consensus.cc:383] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.646796 1181 raft_consensus.cc:738] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.647099 1181 consensus_queue.cc:260] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.645766 996 leader_election.cc:304] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.647792 1181 ts_tablet_manager.cc:1428] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:29.648142 1181 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.649896 2289 raft_consensus.cc:2798] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.649976 2289 raft_consensus.cc:491] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.650043 2289 raft_consensus.cc:3054] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.650059 1181 tablet_bootstrap.cc:654] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.651330 1184 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.653491 996 leader_election.cc:304] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.656083 2735 consensus_queue.cc:1035] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.660296 1184 ts_tablet_manager.cc:1397] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.001s
I20250110 02:51:29.661881 1181 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.661887 1184 raft_consensus.cc:357] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.662045 1181 ts_tablet_manager.cc:1397] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250110 02:51:29.662845 1181 raft_consensus.cc:357] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.663061 1181 raft_consensus.cc:383] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.663169 1181 raft_consensus.cc:738] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.663426 1181 consensus_queue.cc:260] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.664177 1181 ts_tablet_manager.cc:1428] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:29.664532 1181 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.664544 1184 raft_consensus.cc:383] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.664748 1184 raft_consensus.cc:738] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.666056 1181 tablet_bootstrap.cc:654] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.666504 1184 consensus_queue.cc:260] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.667455 1184 ts_tablet_manager.cc:1428] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:29.650619 1188 tablet_bootstrap.cc:492] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:29.647493 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 63505.9us
I20250110 02:51:29.668016 1188 ts_tablet_manager.cc:1397] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.167s user 0.013s sys 0.016s
I20250110 02:51:29.668660 1188 raft_consensus.cc:357] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.668816 1188 raft_consensus.cc:383] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.668867 1057 raft_consensus.cc:2463] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:29.668902 1188 raft_consensus.cc:738] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.669065 1188 consensus_queue.cc:260] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.669257 764 raft_consensus.cc:2463] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:29.669307 711 leader_election.cc:304] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.669476 2024 raft_consensus.cc:2743] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.669562 1188 ts_tablet_manager.cc:1428] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:29.669817 1188 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.671690 1188 tablet_bootstrap.cc:654] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.672951 2199 raft_consensus.cc:2798] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.673063 2199 raft_consensus.cc:695] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:29.673275 2199 consensus_queue.cc:237] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.673594 2357 ts_tablet_manager.cc:1918] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250110 02:51:29.673693 2357 log.cc:1198] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/27a7885a80414f2193400672aa16a911
I20250110 02:51:29.674010 2357 ts_tablet_manager.cc:1939] T 27a7885a80414f2193400672aa16a911 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.676465 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 182) took 5356 ms. Trace:
I20250110 02:51:29.676648 2224 rpcz_store.cc:276] 0110 02:51:24.319794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.319836 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:29.496233 (+5176397us) ts_tablet_manager.cc:1145] Deleting tablet 302af84fd8374ffa96a7503b54849536
0110 02:51:29.496247 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.593395 (+ 97148us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.606506 (+ 13111us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.676450 (+ 69944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5176268}
I20250110 02:51:29.677219 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 302af84fd8374ffa96a7503b54849536 (table default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]) successfully deleted
I20250110 02:51:29.677562 1181 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.677734 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 181) took 5358 ms. Trace:
I20250110 02:51:29.677739 1181 ts_tablet_manager.cc:1397] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20250110 02:51:29.677852 2224 rpcz_store.cc:276] 0110 02:51:24.319709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.319986 (+ 277us) service_pool.cc:225] Handling call
0110 02:51:29.677615 (+5357629us) ts_tablet_manager.cc:1145] Deleting tablet 8d1f2acd39be40e4b71cc0dcf68e7567
0110 02:51:29.677625 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.677720 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5344671}
I20250110 02:51:29.678445 996 leader_election.cc:304] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.678520 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 184) took 5345 ms. Trace:
I20250110 02:51:29.678854 1181 raft_consensus.cc:357] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.679214 2024 raft_consensus.cc:2798] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.679320 2024 raft_consensus.cc:695] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:29.679531 2024 consensus_queue.cc:237] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.679021 1181 raft_consensus.cc:383] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.686542 1181 raft_consensus.cc:738] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.686789 1181 consensus_queue.cc:260] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.687424 1181 ts_tablet_manager.cc:1428] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250110 02:51:29.687736 1181 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.678988 2224 rpcz_store.cc:276] 0110 02:51:24.333308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.335991 (+ 2683us) service_pool.cc:225] Handling call
0110 02:51:29.678439 (+5342448us) ts_tablet_manager.cc:1145] Deleting tablet 8f15078fc22f4b32a48c7190b36e2192
0110 02:51:29.678448 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.678503 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342212}
I20250110 02:51:29.688756 2224 tablet_replica.cc:331] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.689378 2224 raft_consensus.cc:2238] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:29.690277 1181 tablet_bootstrap.cc:654] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.690285 2224 raft_consensus.cc:2267] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.691877 2224 ts_tablet_manager.cc:1905] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.693295 1184 tablet_bootstrap.cc:492] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.703358 1181 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.703466 1181 ts_tablet_manager.cc:1397] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20250110 02:51:29.704038 1181 raft_consensus.cc:357] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.704192 1181 raft_consensus.cc:383] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.704229 1181 raft_consensus.cc:738] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.704361 1181 consensus_queue.cc:260] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.704744 1181 ts_tablet_manager.cc:1428] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:29.704955 1181 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.706604 1184 tablet_bootstrap.cc:654] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.707016 1181 tablet_bootstrap.cc:654] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.710008 1184 tablet_bootstrap.cc:492] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.710182 1184 ts_tablet_manager.cc:1397] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250110 02:51:29.710139 2345 raft_consensus.cc:491] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:29.710296 2345 raft_consensus.cc:513] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.710990 2345 leader_election.cc:290] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.718897 1181 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.719094 1181 ts_tablet_manager.cc:1397] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20250110 02:51:29.719749 1181 raft_consensus.cc:357] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.719975 1181 raft_consensus.cc:383] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.720059 1181 raft_consensus.cc:738] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.720472 2345 raft_consensus.cc:491] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.720319 1181 consensus_queue.cc:260] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.720582 2345 raft_consensus.cc:513] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.720875 1181 ts_tablet_manager.cc:1428] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250110 02:51:29.721042 853 leader_election.cc:343] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:29.721096 1181 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.721124 2345 leader_election.cc:290] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.722522 1181 tablet_bootstrap.cc:654] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:29.723915 853 leader_election.cc:343] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:29.724113 853 leader_election.cc:304] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.724349 2345 raft_consensus.cc:2743] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 02:51:29.725814 853 leader_election.cc:343] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:29.731611 1184 raft_consensus.cc:357] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.731880 1184 raft_consensus.cc:383] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.731941 1184 raft_consensus.cc:738] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.732157 1184 consensus_queue.cc:260] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.732749 1184 ts_tablet_manager.cc:1428] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250110 02:51:29.733081 1184 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.734517 1181 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.734617 1181 ts_tablet_manager.cc:1397] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250110 02:51:29.737735 1184 tablet_bootstrap.cc:654] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.735240 1181 raft_consensus.cc:357] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.746304 1181 raft_consensus.cc:383] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.746459 1181 raft_consensus.cc:738] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.746706 1181 consensus_queue.cc:260] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.747172 2600 consensus_queue.cc:1035] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.748265 1181 ts_tablet_manager.cc:1428] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250110 02:51:29.748610 1181 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.754987 1181 tablet_bootstrap.cc:654] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.762084 2289 raft_consensus.cc:513] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.762737 2289 leader_election.cc:290] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.766201 2224 ts_tablet_manager.cc:1918] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.766296 2224 log.cc:1198] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:29.766607 2224 ts_tablet_manager.cc:1939] T 4d970e2fe72b4727968ca605b3e772cc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.766635 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 137) took 5505 ms. Trace:
I20250110 02:51:29.766736 2357 rpcz_store.cc:276] 0110 02:51:24.261272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.261399 (+ 127us) service_pool.cc:225] Handling call
0110 02:51:29.489675 (+5228276us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:29.489684 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.519946 (+ 30262us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.673497 (+153551us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.766613 (+ 93116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5228128}
I20250110 02:51:29.767374 2357 tablet_replica.cc:331] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.767647 2357 raft_consensus.cc:2238] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.767664 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:29.767874 2357 raft_consensus.cc:2267] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:29.768196 994 leader_election.cc:343] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.768465 2357 ts_tablet_manager.cc:1905] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.768595 1181 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.768716 1181 ts_tablet_manager.cc:1397] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20250110 02:51:29.769454 1181 raft_consensus.cc:357] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.769629 1181 raft_consensus.cc:383] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.769670 1181 raft_consensus.cc:738] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.769910 1181 consensus_queue.cc:260] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.770354 1181 ts_tablet_manager.cc:1428] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:29.770593 1181 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.772614 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:29.772801 774 raft_consensus.cc:3054] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.773200 1181 tablet_bootstrap.cc:654] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.779250 2289 raft_consensus.cc:491] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.779388 2289 raft_consensus.cc:513] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.779989 2289 leader_election.cc:290] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.780359 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.780611 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.780797 764 raft_consensus.cc:2463] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:29.780900 994 leader_election.cc:343] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.781211 996 leader_election.cc:304] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.781495 2289 raft_consensus.cc:2798] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.781561 2289 raft_consensus.cc:491] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.781625 2289 raft_consensus.cc:3054] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.791558 1181 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.791699 1181 ts_tablet_manager.cc:1397] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
W20250110 02:51:29.792436 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
I20250110 02:51:29.792508 1181 raft_consensus.cc:357] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:29.792604 1433 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 114324us
I20250110 02:51:29.792670 1181 raft_consensus.cc:383] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.792711 1181 raft_consensus.cc:738] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.792907 1181 consensus_queue.cc:260] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.793385 1181 ts_tablet_manager.cc:1428] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:29.793375 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 27a7885a80414f2193400672aa16a911 (table default.Table_a4b99ef64b874b609f835a3e58454edd [id=fd26dd454ab24b8abab06cbc13d6192a]) successfully deleted
I20250110 02:51:29.793629 1181 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:29.793601 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because the tablet was not found. No further retry: Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:29.796288 1181 tablet_bootstrap.cc:654] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.797894 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:29.799456 1188 tablet_bootstrap.cc:492] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.799674 1188 ts_tablet_manager.cc:1397] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.130s user 0.010s sys 0.023s
I20250110 02:51:29.800833 1188 raft_consensus.cc:357] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.801057 1188 raft_consensus.cc:383] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.801157 1188 raft_consensus.cc:738] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.801456 1188 consensus_queue.cc:260] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.802099 1188 ts_tablet_manager.cc:1428] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:29.802450 1188 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.803977 2352 ts_tablet_manager.cc:1918] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:29.804117 2352 log.cc:1198] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/89f6797b1076468a93fb0b510c553bed
I20250110 02:51:29.804455 2352 ts_tablet_manager.cc:1939] T 89f6797b1076468a93fb0b510c553bed P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:29.804785 853 leader_election.cc:343] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:29.804927 853 leader_election.cc:304] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:29.805197 2345 raft_consensus.cc:2743] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:29.805912 853 consensus_peers.cc:487] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:29.808151 2456 consensus_queue.cc:1035] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:29.810751 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 179) took 5493 ms. Trace:
I20250110 02:51:29.810845 2224 rpcz_store.cc:276] 0110 02:51:24.317628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.317765 (+ 137us) service_pool.cc:225] Handling call
0110 02:51:29.688712 (+5370947us) ts_tablet_manager.cc:1145] Deleting tablet 4d970e2fe72b4727968ca605b3e772cc
0110 02:51:29.688723 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.762040 (+ 73317us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.766107 (+ 4067us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.810731 (+ 44624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5349681}
I20250110 02:51:29.810940 1188 tablet_bootstrap.cc:654] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.811192 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 183) took 5478 ms. Trace:
I20250110 02:51:29.811256 2224 rpcz_store.cc:276] 0110 02:51:24.333202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.334618 (+ 1416us) service_pool.cc:225] Handling call
0110 02:51:29.811105 (+5476487us) ts_tablet_manager.cc:1145] Deleting tablet 53ae5a6572bd4e7e90e92d4b1cf8d820
0110 02:51:29.811114 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.811178 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22180,"tablet-delete.queue_time_us":5471334}
I20250110 02:51:29.811481 2224 tablet_replica.cc:331] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.811642 2224 raft_consensus.cc:2238] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.811774 2224 raft_consensus.cc:2267] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.812311 2224 ts_tablet_manager.cc:1905] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:29.819116 853 consensus_peers.cc:487] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:29.823922 1184 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.824185 1184 ts_tablet_manager.cc:1397] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.000s
I20250110 02:51:29.824542 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:29.825376 1184 raft_consensus.cc:357] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.825769 1184 raft_consensus.cc:383] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.825902 1184 raft_consensus.cc:738] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.826248 1184 consensus_queue.cc:260] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.827245 1184 ts_tablet_manager.cc:1428] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:29.828042 1184 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.829761 1184 tablet_bootstrap.cc:654] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.834493 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 4d970e2fe72b4727968ca605b3e772cc (table default.Table_5bf91a5e0e004e1880fdf025d5d9ba38 [id=c584ebe680bb449e92795453c9b6ab74]) successfully deleted
W20250110 02:51:29.834856 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because the tablet was not found. No further retry: Not found: Tablet not found: 53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:29.844480 2224 ts_tablet_manager.cc:1918] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.844657 2224 log.cc:1198] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:29.845062 2224 ts_tablet_manager.cc:1939] T 3510e86314144a2ca012aacf3d78eacc P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:29.845348 2289 raft_consensus.cc:513] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.845965 2289 leader_election.cc:290] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.847906 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:29.848417 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5601ca24b8f428e8a51cbeec3664901" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:29.848614 764 raft_consensus.cc:3054] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:29.849273 994 leader_election.cc:343] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.849388 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 152) took 5581 ms. Trace:
I20250110 02:51:29.849496 2352 rpcz_store.cc:276] 0110 02:51:24.267636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.270935 (+ 3299us) service_pool.cc:225] Handling call
0110 02:51:29.476211 (+5205276us) ts_tablet_manager.cc:1145] Deleting tablet 89f6797b1076468a93fb0b510c553bed
0110 02:51:29.476237 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.480742 (+ 4505us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.803872 (+323130us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.849370 (+ 45498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18816,"tablet-delete.queue_time_us":5205187}
I20250110 02:51:29.849960 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 89f6797b1076468a93fb0b510c553bed (table default.Table_1c2495b680684279ae633a18f91a091e [id=e222fd0991c44f5ea885842f04d521c6]) successfully deleted
I20250110 02:51:29.850270 2352 tablet_replica.cc:331] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:29.850481 2352 raft_consensus.cc:2238] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.851024 2352 raft_consensus.cc:2267] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.851578 2352 ts_tablet_manager.cc:1905] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.854481 774 raft_consensus.cc:2463] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:29.856705 996 leader_election.cc:304] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.858160 2600 raft_consensus.cc:2798] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.858309 2600 raft_consensus.cc:695] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:29.858549 2600 consensus_queue.cc:237] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.860141 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:29.860446 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:29.860461 1184 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.860641 1184 ts_tablet_manager.cc:1397] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250110 02:51:29.860693 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:29.860955 1441 catalog_manager.cc:5526] T 7efa857721c34003ad91912ef2d7b381 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:29.862046 1184 raft_consensus.cc:357] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.862355 1441 catalog_manager.cc:5526] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:29.862959 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:29.864135 1181 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.864302 1181 ts_tablet_manager.cc:1397] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.017s
I20250110 02:51:29.865126 1181 raft_consensus.cc:357] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.865365 1181 raft_consensus.cc:383] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.865468 1181 raft_consensus.cc:738] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.865556 1184 raft_consensus.cc:383] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.865712 1184 raft_consensus.cc:738] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.865739 1181 consensus_queue.cc:260] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.866031 1184 consensus_queue.cc:260] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.866482 1181 ts_tablet_manager.cc:1428] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:29.866742 1181 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.866874 1184 ts_tablet_manager.cc:1428] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:29.867328 1184 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.868271 1181 tablet_bootstrap.cc:654] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.873327 1184 tablet_bootstrap.cc:654] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.879227 2777 raft_consensus.cc:491] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.879424 2777 raft_consensus.cc:513] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.880172 1188 tablet_bootstrap.cc:492] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.880304 2777 leader_election.cc:290] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.880333 1188 ts_tablet_manager.cc:1397] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.003s
I20250110 02:51:29.881474 1188 raft_consensus.cc:357] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.881837 1188 raft_consensus.cc:383] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.881727 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:29.882007 1188 raft_consensus.cc:738] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.882349 1188 consensus_queue.cc:260] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.882575 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:29.882766 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:29.882864 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.883543 711 leader_election.cc:343] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:29.883965 709 leader_election.cc:343] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.884052 709 leader_election.cc:304] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.884096 1188 ts_tablet_manager.cc:1428] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250110 02:51:29.884470 2777 raft_consensus.cc:2743] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.891187 1181 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:29.891314 1181 ts_tablet_manager.cc:1397] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20250110 02:51:29.892062 1184 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:29.892019 1181 raft_consensus.cc:357] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.892174 1184 ts_tablet_manager.cc:1397] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20250110 02:51:29.892212 1181 raft_consensus.cc:383] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.892262 1181 raft_consensus.cc:738] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.892475 1181 consensus_queue.cc:260] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.892998 1181 ts_tablet_manager.cc:1428] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:29.893164 1184 raft_consensus.cc:357] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.893378 1184 raft_consensus.cc:383] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.893472 1184 raft_consensus.cc:738] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.893699 1184 consensus_queue.cc:260] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.894408 1184 ts_tablet_manager.cc:1428] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 02:51:29.894693 1184 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:29.896109 1184 tablet_bootstrap.cc:654] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.898874 1181 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:29.900521 1181 tablet_bootstrap.cc:654] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.901366 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 185) took 5557 ms. Trace:
I20250110 02:51:29.901440 2224 rpcz_store.cc:276] 0110 02:51:24.344153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.372191 (+ 28038us) service_pool.cc:225] Handling call
0110 02:51:29.811445 (+5439254us) ts_tablet_manager.cc:1145] Deleting tablet 3510e86314144a2ca012aacf3d78eacc
0110 02:51:29.811453 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.841432 (+ 29979us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.844389 (+ 2957us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.901349 (+ 56960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5439055}
I20250110 02:51:29.901726 2224 tablet_replica.cc:331] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:29.901839 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:29.901888 2224 raft_consensus.cc:2238] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.902027 2224 raft_consensus.cc:2267] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.902235 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:29.902490 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:29.902560 2224 ts_tablet_manager.cc:1905] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.902634 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:29.903134 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:29.903368 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:29.903674 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:29.904197 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:29.904811 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 3510e86314144a2ca012aacf3d78eacc (table default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]) successfully deleted
I20250110 02:51:29.905179 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:29.905375 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:29.906816 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:29.909932 1188 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.911180 1188 tablet_bootstrap.cc:654] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.919246 2777 raft_consensus.cc:491] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.919404 2777 raft_consensus.cc:513] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.920122 2777 leader_election.cc:290] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.920645 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.921149 709 leader_election.cc:343] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.922871 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:29.923372 711 leader_election.cc:343] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.923475 711 leader_election.cc:304] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.923796 2777 raft_consensus.cc:2743] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.934962 1188 tablet_bootstrap.cc:492] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:29.935200 1188 ts_tablet_manager.cc:1397] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.002s
I20250110 02:51:29.936374 1188 raft_consensus.cc:357] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.936667 1188 raft_consensus.cc:383] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:29.936801 1188 raft_consensus.cc:738] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:29.936847 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:29.937361 1188 consensus_queue.cc:260] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.941869 2357 ts_tablet_manager.cc:1918] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:29.941908 1188 ts_tablet_manager.cc:1428] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250110 02:51:29.941979 2357 log.cc:1198] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/676f5f457d9e4028bfd0b707604c42f5
I20250110 02:51:29.942284 1188 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:29.942325 2357 ts_tablet_manager.cc:1939] T 676f5f457d9e4028bfd0b707604c42f5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:29.942528 764 raft_consensus.cc:2463] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:29.947546 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 138) took 5686 ms. Trace:
I20250110 02:51:29.947631 2357 rpcz_store.cc:276] 0110 02:51:24.261364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.261625 (+ 261us) service_pool.cc:225] Handling call
0110 02:51:29.767281 (+5505656us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:29.767295 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.846471 (+ 79176us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.941762 (+ 95291us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:29.947526 (+ 5764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5505517}
I20250110 02:51:29.947923 2357 tablet_replica.cc:331] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:29.948094 2357 raft_consensus.cc:2238] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:29.948228 2357 raft_consensus.cc:2267] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:29.948417 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 676f5f457d9e4028bfd0b707604c42f5 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]) successfully deleted
I20250110 02:51:29.948755 2357 ts_tablet_manager.cc:1905] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:29.949301 1188 tablet_bootstrap.cc:654] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:29.953545 2549 raft_consensus.cc:491] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.953701 2549 raft_consensus.cc:513] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.953915 996 leader_election.cc:304] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.954504 2549 leader_election.cc:290] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:29.956158 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:29.956697 709 leader_election.cc:343] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.958704 2600 raft_consensus.cc:2798] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:29.958866 2600 raft_consensus.cc:695] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:29.959095 2600 consensus_queue.cc:237] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.960638 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:29.962980 711 leader_election.cc:343] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.963124 711 leader_election.cc:304] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:29.963744 2777 raft_consensus.cc:2743] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:29.981335 2289 raft_consensus.cc:491] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.981495 2289 raft_consensus.cc:513] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:29.982100 2289 leader_election.cc:290] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:29.982527 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.982749 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.982944 764 raft_consensus.cc:2463] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:29.983371 994 leader_election.cc:343] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.983686 996 leader_election.cc:304] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.985244 2600 raft_consensus.cc:2798] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.985350 2600 raft_consensus.cc:491] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.985432 2600 raft_consensus.cc:3054] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:29.988365 2456 raft_consensus.cc:491] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:29.988463 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:29.988525 2456 raft_consensus.cc:513] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:29.988701 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:29.989261 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:29.989300 2456 leader_election.cc:290] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:29.989965 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:29.990697 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:29.990916 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:29.991151 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
W20250110 02:51:29.991343 994 leader_election.cc:343] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:29.991394 1441 catalog_manager.cc:5526] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:29.992188 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6598311f59c6476ba339b8e57961cf67 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]). Expected version 1 got 0
I20250110 02:51:29.992645 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:29.992830 764 raft_consensus.cc:2463] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:29.993268 996 leader_election.cc:304] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:29.993516 2289 raft_consensus.cc:2798] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:29.993590 2289 raft_consensus.cc:491] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:29.993654 2289 raft_consensus.cc:3054] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.002203 2777 raft_consensus.cc:491] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.002362 2777 raft_consensus.cc:513] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.002981 2777 leader_election.cc:290] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.003551 1188 tablet_bootstrap.cc:492] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.003743 1188 ts_tablet_manager.cc:1397] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.003s
I20250110 02:51:30.004099 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.004635 711 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.004805 1181 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.004117 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.005051 1181 ts_tablet_manager.cc:1397] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.106s user 0.013s sys 0.021s
I20250110 02:51:30.005144 1188 raft_consensus.cc:357] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:30.005314 709 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.005357 1188 raft_consensus.cc:383] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.005415 1188 raft_consensus.cc:738] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.005424 709 leader_election.cc:304] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.005652 1188 consensus_queue.cc:260] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.006006 2777 raft_consensus.cc:2743] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.006057 1181 raft_consensus.cc:357] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.006330 1181 raft_consensus.cc:383] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.006457 1181 raft_consensus.cc:738] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.006749 1181 consensus_queue.cc:260] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:30.007376 853 consensus_peers.cc:487] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.007876 1181 ts_tablet_manager.cc:1428] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:30.008229 1181 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.009828 1188 ts_tablet_manager.cc:1428] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:30.010568 1188 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.010659 1181 tablet_bootstrap.cc:654] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.017323 1188 tablet_bootstrap.cc:654] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.018589 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:30.018970 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:30.019205 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:30.019341 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:30.019526 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:30.022661 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:30.023479 2777 raft_consensus.cc:491] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.023607 2777 raft_consensus.cc:513] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.024223 2777 leader_election.cc:290] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.024744 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.025385 709 leader_election.cc:343] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.025882 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:30.025952 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.026496 711 leader_election.cc:343] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.026598 711 leader_election.cc:304] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.027005 2777 raft_consensus.cc:2743] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.030797 2224 ts_tablet_manager.cc:1918] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.030898 2224 log.cc:1198] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:30.031262 2224 ts_tablet_manager.cc:1939] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.031483 2600 raft_consensus.cc:513] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.032048 2600 leader_election.cc:290] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.034947 2289 raft_consensus.cc:513] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.035440 2289 leader_election.cc:290] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.039980 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.040359 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.040445 2345 raft_consensus.cc:491] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.040570 2345 raft_consensus.cc:513] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.030503 1184 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.041158 2345 leader_election.cc:290] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.041851 994 leader_election.cc:343] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.041960 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.042039 994 leader_election.cc:343] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.041828 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059fbcbfb7a4f239a5947f702d21c0f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.042734 764 raft_consensus.cc:2388] T 0059fbcbfb7a4f239a5947f702d21c0f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:30.042835 2352 ts_tablet_manager.cc:1918] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.042924 2352 log.cc:1198] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:30.043112 853 leader_election.cc:304] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.043234 2352 ts_tablet_manager.cc:1939] T 32dd1e726f1a41d0894737b3ef3caf8f P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.043463 2345 raft_consensus.cc:3054] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.041357 1184 ts_tablet_manager.cc:1397] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.005s
I20250110 02:51:30.044507 1184 raft_consensus.cc:357] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.044803 1184 raft_consensus.cc:383] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.045189 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 187) took 5682 ms. Trace:
I20250110 02:51:30.045336 1181 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.045459 1181 ts_tablet_manager.cc:1397] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20250110 02:51:30.045271 2224 rpcz_store.cc:276] 0110 02:51:24.362239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.362296 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:29.901681 (+5539385us) ts_tablet_manager.cc:1145] Deleting tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
0110 02:51:29.901690 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.950971 (+ 49281us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.030686 (+ 79715us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.045172 (+ 14486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5528387}
I20250110 02:51:30.045717 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 186) took 5683 ms. Trace:
I20250110 02:51:30.045789 2224 rpcz_store.cc:276] 0110 02:51:24.362136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.362326 (+ 190us) service_pool.cc:225] Handling call
0110 02:51:30.045619 (+5683293us) ts_tablet_manager.cc:1145] Deleting tablet 28340f0f9a2043a199c74f13755eb702
0110 02:51:30.045630 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.045702 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5662230}
I20250110 02:51:30.046099 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 193) took 5656 ms. Trace:
I20250110 02:51:30.046165 2224 rpcz_store.cc:276] 0110 02:51:24.389272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.389315 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:30.045995 (+5656680us) ts_tablet_manager.cc:1145] Deleting tablet 82bd4307e5fe4f61b9308005cf3446e1
0110 02:51:30.046004 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.046085 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656516}
I20250110 02:51:30.046255 1181 raft_consensus.cc:357] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.046463 1181 raft_consensus.cc:383] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.046499 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 188) took 5665 ms. Trace:
I20250110 02:51:30.046520 1181 raft_consensus.cc:738] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.046557 2224 rpcz_store.cc:276] 0110 02:51:24.381277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.381764 (+ 487us) service_pool.cc:225] Handling call
0110 02:51:30.046382 (+5664618us) ts_tablet_manager.cc:1145] Deleting tablet f9b379b357194d729108428cbaebd51a
0110 02:51:30.046391 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.046485 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5655527}
I20250110 02:51:30.046787 2224 tablet_replica.cc:331] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.046715 1181 consensus_queue.cc:260] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.046969 2224 raft_consensus.cc:2238] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.047103 2224 raft_consensus.cc:2267] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.047209 1181 ts_tablet_manager.cc:1428] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:30.047456 1181 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.047601 2224 ts_tablet_manager.cc:1905] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.050671 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]) successfully deleted
W20250110 02:51:30.050976 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because the tablet was not found. No further retry: Not found: Tablet not found: 28340f0f9a2043a199c74f13755eb702
W20250110 02:51:30.051203 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
W20250110 02:51:30.051471 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:30.051906 1188 tablet_bootstrap.cc:492] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.052058 1188 ts_tablet_manager.cc:1397] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
W20250110 02:51:30.052732 853 consensus_peers.cc:487] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.046370 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08705d260c0a462b94ba6b0dc5121123" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.052822 1188 raft_consensus.cc:357] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.053014 1188 raft_consensus.cc:383] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.053062 1188 raft_consensus.cc:738] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.053050 774 raft_consensus.cc:3054] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.053385 1188 consensus_queue.cc:260] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.053892 1188 ts_tablet_manager.cc:1428] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:30.054329 1188 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.046936 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.055480 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:30.055511 764 raft_consensus.cc:3054] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.055708 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:30.056051 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:30.056320 1181 tablet_bootstrap.cc:654] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.056452 1184 raft_consensus.cc:738] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.059028 2777 consensus_queue.cc:1035] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.066655 1184 consensus_queue.cc:260] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.068116 1184 ts_tablet_manager.cc:1428] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250110 02:51:30.068154 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 361c332d66984cf7ab84d74e913a5bde (table default.Table_5f2cdcc0d6c94163b11c6639fdf0b9d5 [id=0692620395c14593803592e6aade6af3]). Expected version 3 got 2
I20250110 02:51:30.068620 1441 catalog_manager.cc:5526] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:30.069382 1441 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_6aaaf7ba453a4e74a3a55367388c0584 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 3 got 0
W20250110 02:51:30.073397 709 consensus_peers.cc:487] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.077093 1181 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.077222 1181 ts_tablet_manager.cc:1397] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.007s
I20250110 02:51:30.077883 1181 raft_consensus.cc:357] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.078055 1181 raft_consensus.cc:383] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.078114 1181 raft_consensus.cc:738] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.078334 1181 consensus_queue.cc:260] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.078925 1181 ts_tablet_manager.cc:1428] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:30.079169 1181 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.080979 1181 tablet_bootstrap.cc:654] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.080981 915 raft_consensus.cc:3054] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.081216 1184 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.085163 2357 ts_tablet_manager.cc:1918] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.085261 2357 log.cc:1198] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:30.085629 2357 ts_tablet_manager.cc:1939] T d4f1d916a5ea48708c2c1a61c92fe137 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.085861 2289 raft_consensus.cc:491] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.086005 2289 raft_consensus.cc:513] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.086742 2289 leader_election.cc:290] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.087388 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.088217 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:30.088510 994 leader_election.cc:343] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.088565 767 raft_consensus.cc:2463] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.088940 996 leader_election.cc:304] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.089483 2289 raft_consensus.cc:2798] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.089587 2289 raft_consensus.cc:491] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.089655 2289 raft_consensus.cc:3054] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.091413 1181 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.091529 1181 ts_tablet_manager.cc:1397] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20250110 02:51:30.092173 1181 raft_consensus.cc:357] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.092340 1181 raft_consensus.cc:383] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.092394 1181 raft_consensus.cc:738] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.092554 1181 consensus_queue.cc:260] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.092944 1181 ts_tablet_manager.cc:1428] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:30.093155 1181 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.094481 1181 tablet_bootstrap.cc:654] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.095324 1184 tablet_bootstrap.cc:654] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:30.096493 996 consensus_peers.cc:487] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 676f5f457d9e4028bfd0b707604c42f5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.098708 1188 tablet_bootstrap.cc:654] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.101898 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 174) took 5832 ms. Trace:
I20250110 02:51:30.101979 2352 rpcz_store.cc:276] 0110 02:51:24.269899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.271054 (+ 1155us) service_pool.cc:225] Handling call
0110 02:51:29.850224 (+5579170us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:29.850235 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:29.955266 (+105031us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.042734 (+ 87468us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.101880 (+ 59146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5579076}
I20250110 02:51:30.102284 2352 tablet_replica.cc:331] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.102481 1181 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.102499 2352 raft_consensus.cc:2238] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.102587 1181 ts_tablet_manager.cc:1397] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20250110 02:51:30.102654 2352 raft_consensus.cc:2267] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.103139 2352 ts_tablet_manager.cc:1905] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.103240 1181 raft_consensus.cc:357] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.103403 1181 raft_consensus.cc:383] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.103446 1181 raft_consensus.cc:738] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.103610 1181 consensus_queue.cc:260] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.104409 1181 ts_tablet_manager.cc:1428] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:30.104637 1181 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.106052 1181 tablet_bootstrap.cc:654] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.106554 2345 raft_consensus.cc:2743] T 0059fbcbfb7a4f239a5947f702d21c0f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.110626 774 raft_consensus.cc:2463] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.113291 1184 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.113472 1184 ts_tablet_manager.cc:1397] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250110 02:51:30.114586 1184 raft_consensus.cc:357] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.114838 1184 raft_consensus.cc:383] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.114912 1184 raft_consensus.cc:738] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.115180 1184 consensus_queue.cc:260] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.115998 1184 ts_tablet_manager.cc:1428] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:30.116417 1184 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.118505 2600 raft_consensus.cc:491] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.118705 2600 raft_consensus.cc:513] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.119609 2600 leader_election.cc:290] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.120106 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 32dd1e726f1a41d0894737b3ef3caf8f (table default.Table_99c8d182fc924995a8f6e78bd31f07a6 [id=5cf31203f20a44c4b0653bc28978da0b]) successfully deleted
I20250110 02:51:30.121070 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.121079 996 leader_election.cc:304] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
W20250110 02:51:30.121595 994 leader_election.cc:343] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.121651 2456 raft_consensus.cc:2798] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.121775 2456 raft_consensus.cc:695] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.121992 2456 consensus_queue.cc:237] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.122666 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.122872 767 raft_consensus.cc:2463] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.124926 2024 raft_consensus.cc:491] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.124646 1184 tablet_bootstrap.cc:654] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.125078 2024 raft_consensus.cc:513] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.125775 2024 leader_election.cc:290] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.126155 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.126260 996 leader_election.cc:304] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
W20250110 02:51:30.126731 994 leader_election.cc:343] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.126842 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.127017 767 raft_consensus.cc:2463] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.127945 996 leader_election.cc:304] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.128515 2478 raft_consensus.cc:2798] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.128629 2478 raft_consensus.cc:491] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.128751 2478 raft_consensus.cc:3054] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.129446 2735 raft_consensus.cc:2798] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.129556 2735 raft_consensus.cc:491] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.129647 2735 raft_consensus.cc:3054] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.130443 2289 raft_consensus.cc:513] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.131155 2289 leader_election.cc:290] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.131531 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.132084 994 leader_election.cc:343] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.134827 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3858638b6fc48d0bb8b8f97370a791d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.135007 767 raft_consensus.cc:3054] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.136004 2289 raft_consensus.cc:491] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.136240 1188 tablet_bootstrap.cc:492] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.136431 1188 ts_tablet_manager.cc:1397] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.005s
I20250110 02:51:30.136219 2289 raft_consensus.cc:513] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.136988 2289 leader_election.cc:290] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.137326 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.137526 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.137562 1188 raft_consensus.cc:357] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.137710 774 raft_consensus.cc:2463] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.137753 1188 raft_consensus.cc:383] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.137812 1188 raft_consensus.cc:738] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:30.137964 994 leader_election.cc:343] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.138235 996 leader_election.cc:304] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.138232 1188 consensus_queue.cc:260] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.138913 1188 ts_tablet_manager.cc:1428] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:30.139683 2289 raft_consensus.cc:2798] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.139775 2289 raft_consensus.cc:491] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.139833 2289 raft_consensus.cc:3054] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.142213 1188 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.143828 1188 tablet_bootstrap.cc:654] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.151252 2456 raft_consensus.cc:491] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.151400 2456 raft_consensus.cc:513] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.152164 2456 leader_election.cc:290] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.152630 853 consensus_peers.cc:487] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.152664 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.153141 994 leader_election.cc:343] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.153383 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.153635 774 raft_consensus.cc:2463] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.158751 996 leader_election.cc:304] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.159344 2456 raft_consensus.cc:2798] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.159438 2456 raft_consensus.cc:491] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.159526 2456 raft_consensus.cc:3054] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.161932 1184 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.162565 2549 raft_consensus.cc:491] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.162686 2549 raft_consensus.cc:513] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.163214 2549 leader_election.cc:290] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.163842 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.164402 709 leader_election.cc:343] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.165740 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.166256 711 leader_election.cc:343] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.166378 711 leader_election.cc:304] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.167783 2549 raft_consensus.cc:2743] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.165817 1184 ts_tablet_manager.cc:1397] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20250110 02:51:30.174427 1188 tablet_bootstrap.cc:492] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.174623 1188 ts_tablet_manager.cc:1397] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.009s
I20250110 02:51:30.175347 2738 raft_consensus.cc:491] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.175470 2738 raft_consensus.cc:513] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.175467 1188 raft_consensus.cc:357] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.175633 1188 raft_consensus.cc:383] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.175627 2549 raft_consensus.cc:491] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.175679 1188 raft_consensus.cc:738] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.175751 2549 raft_consensus.cc:513] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.175894 1188 consensus_queue.cc:260] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.176085 2738 leader_election.cc:290] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.176370 2549 leader_election.cc:290] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.176465 1188 ts_tablet_manager.cc:1428] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:30.176728 1188 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.178165 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.178257 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.178408 774 raft_consensus.cc:2463] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:30.179275 711 leader_election.cc:343] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.176934 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.177501 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.179605 996 leader_election.cc:304] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.179955 2738 raft_consensus.cc:2798] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.180052 2738 raft_consensus.cc:491] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.180111 2738 raft_consensus.cc:3054] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:30.180176 709 leader_election.cc:343] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.180285 709 leader_election.cc:304] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:30.180742 994 leader_election.cc:343] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.180842 2549 raft_consensus.cc:2743] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.181455 1188 tablet_bootstrap.cc:654] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.185809 2024 raft_consensus.cc:491] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.185933 2024 raft_consensus.cc:513] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.186339 1181 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.186475 1181 ts_tablet_manager.cc:1397] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.018s
I20250110 02:51:30.186537 2024 leader_election.cc:290] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.187094 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.187230 1181 raft_consensus.cc:357] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.187302 774 raft_consensus.cc:2463] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.187392 1181 raft_consensus.cc:383] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.187440 1181 raft_consensus.cc:738] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.187610 1181 consensus_queue.cc:260] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.187755 996 leader_election.cc:304] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.188097 1181 ts_tablet_manager.cc:1428] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:30.188138 2600 raft_consensus.cc:2798] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.188212 2600 raft_consensus.cc:491] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.188282 2600 raft_consensus.cc:3054] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.188378 1181 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.189007 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.189277 2378 raft_consensus.cc:491] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.189419 2378 raft_consensus.cc:513] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:30.189700 994 leader_election.cc:343] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.189965 1181 tablet_bootstrap.cc:654] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.190124 2378 leader_election.cc:290] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.190734 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.190925 774 raft_consensus.cc:2388] T 8bc32061409246a2b280dde4f9180a8e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:30.192610 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc32061409246a2b280dde4f9180a8e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.193094 853 leader_election.cc:304] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.193084 2024 consensus_queue.cc:1035] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.193501 2345 raft_consensus.cc:3054] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.194988 2549 raft_consensus.cc:491] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.195163 2549 raft_consensus.cc:513] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.195889 2549 leader_election.cc:290] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.196520 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.197101 709 leader_election.cc:343] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.197808 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.198374 711 leader_election.cc:343] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.198496 711 leader_election.cc:304] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.198838 1181 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.198907 2549 raft_consensus.cc:2743] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.199023 1181 ts_tablet_manager.cc:1397] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20250110 02:51:30.199846 1181 raft_consensus.cc:357] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.200016 1181 raft_consensus.cc:383] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.200068 1181 raft_consensus.cc:738] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.200246 1181 consensus_queue.cc:260] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.202723 2352 ts_tablet_manager.cc:1918] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.202822 2352 log.cc:1198] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/95b14d7592024a34baf7756d92e546d9
I20250110 02:51:30.202227 1184 raft_consensus.cc:357] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.203186 2352 ts_tablet_manager.cc:1939] T 95b14d7592024a34baf7756d92e546d9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.203299 1184 raft_consensus.cc:383] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.203699 1184 raft_consensus.cc:738] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.204805 1184 consensus_queue.cc:260] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.208364 764 raft_consensus.cc:2463] T 84adc1fd3f76497b98a0fcbc52a3120a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.208822 2478 raft_consensus.cc:513] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.209563 2478 leader_election.cc:290] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.211097 2224 ts_tablet_manager.cc:1918] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.211228 2224 log.cc:1198] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d424b4da66184bd19eccefad80621516
I20250110 02:51:30.211576 2224 ts_tablet_manager.cc:1939] T d424b4da66184bd19eccefad80621516 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.212419 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 139) took 5950 ms. Trace:
I20250110 02:51:30.212499 2357 rpcz_store.cc:276] 0110 02:51:24.262001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.262037 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:29.947881 (+5685844us) ts_tablet_manager.cc:1145] Deleting tablet d4f1d916a5ea48708c2c1a61c92fe137
0110 02:51:29.947892 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.036596 (+ 88704us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.085053 (+ 48457us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.212401 (+127348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5685664}
I20250110 02:51:30.212852 2357 tablet_replica.cc:331] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.215342 1184 ts_tablet_manager.cc:1428] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20250110 02:51:30.217499 1184 tablet_bootstrap.cc:492] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.220722 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.220804 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 175) took 5949 ms. Trace:
I20250110 02:51:30.220867 917 raft_consensus.cc:3054] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.220881 2352 rpcz_store.cc:276] 0110 02:51:24.271587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.271630 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:30.102238 (+5830608us) ts_tablet_manager.cc:1145] Deleting tablet 95b14d7592024a34baf7756d92e546d9
0110 02:51:30.102249 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.106150 (+ 3901us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.202595 (+ 96445us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.220789 (+ 18194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5830480}
I20250110 02:51:30.223172 2352 tablet_replica.cc:331] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.223351 2352 raft_consensus.cc:2238] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.223474 2352 raft_consensus.cc:2267] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.223946 2352 ts_tablet_manager.cc:1905] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.225035 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d4f1d916a5ea48708c2c1a61c92fe137 (table default.Table_48581fba686f44bea7a67b12a11e149f [id=e856d2ec8a6c4e2793bd34b38dbe8a05]) successfully deleted
I20250110 02:51:30.225135 2357 raft_consensus.cc:2238] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.225292 2357 raft_consensus.cc:2267] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.225759 2357 ts_tablet_manager.cc:1905] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.226568 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 95b14d7592024a34baf7756d92e546d9 (table default.Table_bf5730ffdb414255925a1360ec7d7626 [id=8dafaf90f6e7460284b6fc3434eed1e4]) successfully deleted
I20250110 02:51:30.229964 2691 raft_consensus.cc:491] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.230120 2691 raft_consensus.cc:513] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.230881 2691 leader_election.cc:290] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.231302 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.231570 1061 raft_consensus.cc:2388] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:30.232568 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00698a0a9ee4e5da69a745f4a3aa8b4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.232795 764 raft_consensus.cc:2388] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:30.233246 853 leader_election.cc:304] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.233677 2691 raft_consensus.cc:3054] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.234746 1184 tablet_bootstrap.cc:654] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.242256 2478 raft_consensus.cc:491] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.242391 2478 raft_consensus.cc:513] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.242579 996 leader_election.cc:304] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.242967 2478 leader_election.cc:290] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.243101 2478 raft_consensus.cc:2798] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.243178 2478 raft_consensus.cc:695] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.243381 2478 consensus_queue.cc:237] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.244820 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.245394 994 leader_election.cc:343] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.246863 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd120d8edf14151b7ebed3b74e2b399" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.247046 764 raft_consensus.cc:3054] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.247108 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.247309 768 raft_consensus.cc:2463] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.250536 2378 raft_consensus.cc:491] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.250674 2378 raft_consensus.cc:513] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.251338 2378 leader_election.cc:290] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.252362 853 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:30.252643 996 leader_election.cc:304] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.254487 2549 raft_consensus.cc:491] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.254632 2549 raft_consensus.cc:513] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.255299 2549 leader_election.cc:290] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.260299 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.273797 2906 raft_consensus.cc:491] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.273996 2906 raft_consensus.cc:513] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.275103 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.275545 994 leader_election.cc:343] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.275754 2906 leader_election.cc:290] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.276930 2906 raft_consensus.cc:2798] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.277058 2906 raft_consensus.cc:491] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.277166 2906 raft_consensus.cc:3054] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.283278 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.283458 1061 raft_consensus.cc:2463] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:30.284060 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.284224 1061 raft_consensus.cc:2463] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:30.284737 2898 raft_consensus.cc:491] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.284870 2898 raft_consensus.cc:513] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.285773 1181 ts_tablet_manager.cc:1428] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.087s user 0.004s sys 0.023s
I20250110 02:51:30.285852 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.286110 1181 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:30.286335 994 leader_election.cc:343] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.287205 711 leader_election.cc:304] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.287490 2549 raft_consensus.cc:2798] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.287550 2549 raft_consensus.cc:491] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.287596 2549 raft_consensus.cc:3054] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.288043 853 leader_election.cc:304] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.288272 1181 tablet_bootstrap.cc:654] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.288656 2378 raft_consensus.cc:2798] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.288740 2378 raft_consensus.cc:491] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.288796 2378 raft_consensus.cc:3054] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.240006 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:30.289336 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:30.290199 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:30.290578 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:30.290877 2199 raft_consensus.cc:491] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.290935 1181 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.291059 1181 ts_tablet_manager.cc:1397] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250110 02:51:30.291014 2199 raft_consensus.cc:513] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.291474 2898 leader_election.cc:290] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.291543 2199 leader_election.cc:290] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.292203 1181 raft_consensus.cc:357] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.292301 1188 tablet_bootstrap.cc:492] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.292382 1181 raft_consensus.cc:383] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.292402 1188 ts_tablet_manager.cc:1397] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.005s
I20250110 02:51:30.292472 1181 raft_consensus.cc:738] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.292505 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:30.292750 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:30.292676 1181 consensus_queue.cc:260] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.293138 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:30.293200 1181 ts_tablet_manager.cc:1428] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:30.293095 1188 raft_consensus.cc:357] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.293279 1188 raft_consensus.cc:383] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.293350 1188 raft_consensus.cc:738] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.293466 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:30.293545 1181 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.293506 1188 consensus_queue.cc:260] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.293773 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:30.293987 1188 ts_tablet_manager.cc:1428] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:30.294104 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:30.294234 1188 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.294384 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:30.294584 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:30.294958 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:30.295224 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:30.295449 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:30.295719 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:30.295791 1181 tablet_bootstrap.cc:654] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.295974 1188 tablet_bootstrap.cc:654] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.296104 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:30.296571 767 raft_consensus.cc:2463] T d3858638b6fc48d0bb8b8f97370a791d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.296877 2345 raft_consensus.cc:2743] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.298095 2289 raft_consensus.cc:513] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.298688 2289 leader_election.cc:290] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.298930 764 raft_consensus.cc:2463] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.300559 2549 raft_consensus.cc:513] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.301070 2549 leader_election.cc:290] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.301398 917 raft_consensus.cc:2463] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.301913 994 leader_election.cc:304] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:30.304340 2691 raft_consensus.cc:2743] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.305177 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.305600 994 leader_election.cc:343] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.306622 2735 raft_consensus.cc:513] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.306739 2289 raft_consensus.cc:2798] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.306833 2289 raft_consensus.cc:695] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
W20250110 02:51:30.306914 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 54296.8us
I20250110 02:51:30.307039 2289 consensus_queue.cc:237] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.307140 2735 leader_election.cc:290] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.307304 996 leader_election.cc:304] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.309480 2352 ts_tablet_manager.cc:1918] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.309569 2352 log.cc:1198] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:30.309824 2352 ts_tablet_manager.cc:1939] T 6e4d3270e3a04cc99f90943b9573ec7e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.310103 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 176) took 6038 ms. Trace:
I20250110 02:51:30.310168 2352 rpcz_store.cc:276] 0110 02:51:24.271994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.272030 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:30.223130 (+5951100us) ts_tablet_manager.cc:1145] Deleting tablet 6e4d3270e3a04cc99f90943b9573ec7e
0110 02:51:30.223139 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.296532 (+ 73393us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.309391 (+ 12859us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.310091 (+ 700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5950971}
I20250110 02:51:30.310847 2378 raft_consensus.cc:513] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.311894 2899 raft_consensus.cc:491] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.311996 2899 raft_consensus.cc:513] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.312474 2899 leader_election.cc:290] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.312691 2899 consensus_queue.cc:1035] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.313048 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.313140 2899 raft_consensus.cc:2798] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.313225 2899 raft_consensus.cc:695] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.313385 2899 consensus_queue.cc:237] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.314538 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.314725 764 raft_consensus.cc:2463] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.314960 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.315114 2352 tablet_replica.cc:331] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.315128 770 raft_consensus.cc:2463] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.315182 2738 raft_consensus.cc:513] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.315292 2352 raft_consensus.cc:2238] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.315348 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.315414 2352 raft_consensus.cc:2267] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.315485 774 raft_consensus.cc:3054] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.315616 2738 leader_election.cc:290] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.315834 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.315912 2352 ts_tablet_manager.cc:1905] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.315963 768 raft_consensus.cc:3054] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.318334 774 raft_consensus.cc:2463] T c52f74fe375440a28139c27f3f868970 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.314395 1184 tablet_bootstrap.cc:492] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.319705 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.320178 994 leader_election.cc:343] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.320863 1184 ts_tablet_manager.cc:1397] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.000s
I20250110 02:51:30.320963 2456 raft_consensus.cc:513] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.321580 2456 leader_election.cc:290] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.322494 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 189) took 5940 ms. Trace:
I20250110 02:51:30.322587 2224 rpcz_store.cc:276] 0110 02:51:24.382345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.383652 (+ 1307us) service_pool.cc:225] Handling call
0110 02:51:30.046748 (+5663096us) ts_tablet_manager.cc:1145] Deleting tablet d424b4da66184bd19eccefad80621516
0110 02:51:30.046756 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.106854 (+ 60098us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.210977 (+104123us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.322473 (+111496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5647516}
I20250110 02:51:30.322979 2224 tablet_replica.cc:331] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.323225 2224 raft_consensus.cc:2238] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:30.323585 2224 raft_consensus.cc:2267] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.324162 2224 ts_tablet_manager.cc:1905] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:30.325906 711 leader_election.cc:343] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.325932 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d424b4da66184bd19eccefad80621516 (table default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]) successfully deleted
I20250110 02:51:30.325081 1184 raft_consensus.cc:357] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.326308 1184 raft_consensus.cc:383] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.326781 1184 raft_consensus.cc:738] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.327446 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:30.327533 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:30.327694 1057 raft_consensus.cc:3054] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.328495 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:30.328872 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80186dd7d1a240068584b64ce9e9b03f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:30.329021 1061 raft_consensus.cc:3054] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.328169 1184 consensus_queue.cc:260] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.330117 1057 raft_consensus.cc:2463] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:30.331229 1061 raft_consensus.cc:2463] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:30.333232 853 leader_election.cc:304] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:30.333336 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:30.333482 2908 raft_consensus.cc:2798] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.333581 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:30.333618 2908 raft_consensus.cc:695] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:30.333739 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:30.334465 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:30.335438 996 leader_election.cc:304] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.335618 2905 raft_consensus.cc:491] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.336270 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.336442 770 raft_consensus.cc:2463] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.336374 2905 raft_consensus.cc:513] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.337330 2456 raft_consensus.cc:2798] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.337404 2456 raft_consensus.cc:491] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.337455 2456 raft_consensus.cc:3054] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.338933 2357 ts_tablet_manager.cc:1918] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.339025 2357 log.cc:1198] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:30.339259 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71534cc7dd014bc1afb365de96977f8c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.339388 2357 ts_tablet_manager.cc:1939] T b43bb790e85343308a1bb1e1110bfb9c P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.339432 764 raft_consensus.cc:3054] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.339650 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 140) took 6077 ms. Trace:
I20250110 02:51:30.339720 2357 rpcz_store.cc:276] 0110 02:51:24.262408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.262449 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:30.212800 (+5950351us) ts_tablet_manager.cc:1145] Deleting tablet b43bb790e85343308a1bb1e1110bfb9c
0110 02:51:30.212812 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.298942 (+ 86130us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.338828 (+ 39886us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.339634 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12089,"tablet-delete.queue_time_us":5937557}
I20250110 02:51:30.339809 2456 raft_consensus.cc:513] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.340170 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet b43bb790e85343308a1bb1e1110bfb9c (table default.Table_83b75cdbc7424f579e0297ecd7b0d4a4 [id=27a2d816626544e68abffc59b5980171]) successfully deleted
I20250110 02:51:30.340453 2357 tablet_replica.cc:331] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.340567 2456 leader_election.cc:290] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.340648 2357 raft_consensus.cc:2238] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:30.340953 2357 raft_consensus.cc:2267] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.340930 996 leader_election.cc:304] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.341193 2905 leader_election.cc:290] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.342139 996 leader_election.cc:304] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.342235 2357 ts_tablet_manager.cc:1905] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.342279 764 raft_consensus.cc:2463] T 71534cc7dd014bc1afb365de96977f8c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.342784 711 leader_election.cc:304] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.342330 2905 raft_consensus.cc:2798] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.343164 2905 raft_consensus.cc:491] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.343250 2549 raft_consensus.cc:2798] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.343276 2905 raft_consensus.cc:3054] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.343436 2549 raft_consensus.cc:695] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:30.343709 2549 consensus_queue.cc:237] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.344063 2456 raft_consensus.cc:2798] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.344254 2456 raft_consensus.cc:695] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.344377 768 raft_consensus.cc:2463] T 8c7fbdab52014dddbf63cfef69b09b7b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.344493 2456 consensus_queue.cc:237] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.345078 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.345278 764 raft_consensus.cc:3054] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.346657 2378 leader_election.cc:290] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.347596 853 leader_election.cc:343] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:30.348109 764 raft_consensus.cc:2463] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.357263 996 leader_election.cc:304] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.357520 2738 raft_consensus.cc:2798] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.357576 2738 raft_consensus.cc:491] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.357637 2738 raft_consensus.cc:3054] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.359850 2738 raft_consensus.cc:513] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.360658 2906 raft_consensus.cc:513] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.361189 2738 leader_election.cc:290] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.362440 1184 ts_tablet_manager.cc:1428] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20250110 02:51:30.364145 2905 raft_consensus.cc:513] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.364678 2224 ts_tablet_manager.cc:1918] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.365114 1184 tablet_bootstrap.cc:492] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.366449 2224 log.cc:1198] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/971f97a3be9b49958b1218a275e126d4
I20250110 02:51:30.366843 2224 ts_tablet_manager.cc:1939] T 971f97a3be9b49958b1218a275e126d4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.367161 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 201) took 5963 ms. Trace:
I20250110 02:51:30.367249 996 leader_election.cc:304] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.368291 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:30.368726 1181 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.367275 2224 rpcz_store.cc:276] 0110 02:51:24.403438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.405195 (+ 1757us) service_pool.cc:225] Handling call
0110 02:51:30.322930 (+5917735us) ts_tablet_manager.cc:1145] Deleting tablet 971f97a3be9b49958b1218a275e126d4
0110 02:51:30.322940 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.362122 (+ 39182us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.364584 (+ 2462us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.367146 (+ 2562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5917589}
I20250110 02:51:30.368762 996 leader_election.cc:304] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.369266 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:30.369609 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 208) took 5445 ms. Trace:
I20250110 02:51:30.369743 2224 rpcz_store.cc:276] 0110 02:51:24.924498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.924615 (+ 117us) service_pool.cc:225] Handling call
0110 02:51:30.369473 (+5444858us) ts_tablet_manager.cc:1145] Deleting tablet b45b91d084a44c1e8cdf1553967878e2
0110 02:51:30.369486 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.369591 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5438304}
I20250110 02:51:30.370035 2738 raft_consensus.cc:2798] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.368868 2456 raft_consensus.cc:2798] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.370397 2738 raft_consensus.cc:695] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.370877 2738 consensus_queue.cc:237] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.370911 1181 ts_tablet_manager.cc:1397] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.011s
I20250110 02:51:30.371975 2549 raft_consensus.cc:491] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.372154 2549 raft_consensus.cc:513] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.372321 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 971f97a3be9b49958b1218a275e126d4 (table default.Table_c8a984ce6fa04f7687f873fff9a693d1 [id=112bedc144dd49d2ab2fe7e4b541f208]) successfully deleted
W20250110 02:51:30.372663 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b45b91d084a44c1e8cdf1553967878e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b45b91d084a44c1e8cdf1553967878e2 already in progress: creating tablet
I20250110 02:51:30.373021 2549 leader_election.cc:290] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.373014 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.370535 2456 raft_consensus.cc:695] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.370106 2224 tablet_replica.cc:331] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.373322 774 raft_consensus.cc:3054] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.373586 2224 raft_consensus.cc:2238] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:30.373857 2456 consensus_queue.cc:237] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.374650 2224 raft_consensus.cc:2267] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.375905 774 raft_consensus.cc:2463] T ab302fedec4f4c03989b5d3932dc8d4c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.361189 2906 leader_election.cc:290] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.376564 996 leader_election.cc:304] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.377288 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:30.377553 2224 ts_tablet_manager.cc:1905] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.377758 2738 raft_consensus.cc:2798] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.377900 2738 raft_consensus.cc:695] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.378132 2738 consensus_queue.cc:237] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:30.379633 996 leader_election.cc:343] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:30.379930 1181 raft_consensus.cc:357] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.380923 1181 raft_consensus.cc:383] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.381201 1181 raft_consensus.cc:738] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.381371 996 leader_election.cc:304] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.382236 2456 raft_consensus.cc:2798] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.382393 2456 raft_consensus.cc:695] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.382652 2456 consensus_queue.cc:237] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.380661 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:30.383893 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:30.384878 1184 tablet_bootstrap.cc:654] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.385242 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.385409 764 raft_consensus.cc:3054] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.387712 764 raft_consensus.cc:2463] T 39cfc508c1b84299a2f16d307bb25354 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.388310 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.388479 770 raft_consensus.cc:3054] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.388623 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:30.389101 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6598311f59c6476ba339b8e57961cf67 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]). Expected version 1 got 0
I20250110 02:51:30.389441 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:30.390352 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:30.390547 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:30.385637 1181 consensus_queue.cc:260] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.392954 1181 ts_tablet_manager.cc:1428] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250110 02:51:30.394482 1181 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.398712 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_6aaaf7ba453a4e74a3a55367388c0584 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 3 got 0
I20250110 02:51:30.399372 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:30.400151 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:30.401083 1439 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:30.402685 1184 tablet_bootstrap.cc:492] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.403133 2224 ts_tablet_manager.cc:1918] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.403223 2224 log.cc:1198] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/1edb4e06e40c43438882e1eb622d788e
I20250110 02:51:30.403455 1184 ts_tablet_manager.cc:1397] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20250110 02:51:30.403975 1181 tablet_bootstrap.cc:654] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.404629 770 raft_consensus.cc:2463] T b010e9c549214937889fa281542e752e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.405555 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.405779 774 raft_consensus.cc:3054] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.405977 2224 ts_tablet_manager.cc:1939] T 1edb4e06e40c43438882e1eb622d788e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.406392 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 210) took 5442 ms. Trace:
I20250110 02:51:30.408334 774 raft_consensus.cc:2463] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.408741 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 3ca895eca14748c19fdb7d363499b544
I20250110 02:51:30.408072 1184 raft_consensus.cc:357] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.409318 1450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 159) took 1619 ms. Trace:
I20250110 02:51:30.409356 1184 raft_consensus.cc:383] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.409745 1184 raft_consensus.cc:738] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.409432 1450 rpcz_store.cc:276] 0110 02:51:28.790002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.466416 (+676414us) service_pool.cc:225] Handling call
0110 02:51:29.504891 (+ 38475us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:29.505041 (+ 150us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:29.505103 (+ 62us) catalog_manager.cc:2762] Locking tablets
0110 02:51:30.190575 (+685472us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:30.207590 (+ 17015us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:30.207696 (+ 106us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:30.408664 (+200968us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:30.409277 (+ 613us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:30.191464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:30.191579 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:30.191586 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:30.191590 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:30.191713 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:30.191727 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:30.191731 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:30.191843 (+ 112us) tablet.cc:698] Row locks acquired
0110 02:51:30.191847 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:30.191881 (+ 34us) write_op.cc:270] Start()
0110 02:51:30.191912 (+ 31us) write_op.cc:276] Timestamp: P: 1736477490191848 usec, L: 0
0110 02:51:30.191916 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:30.192095 (+ 179us) log.cc:844] Serialized 759 byte log entry
0110 02:51:30.206709 (+ 14614us) op_driver.cc:464] REPLICATION: finished
0110 02:51:30.206831 (+ 122us) write_op.cc:301] APPLY: starting
0110 02:51:30.206849 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:30.206902 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:30.206927 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:30.207013 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:30.207048 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:30.207055 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:30.207134 (+ 79us) log.cc:844] Serialized 29 byte log entry
0110 02:51:30.207196 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.207217 (+ 21us) write_op.cc:454] Released schema lock
0110 02:51:30.207224 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:30.207309 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.207311 (+ 2us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":171,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":725,"replication_time_us":14762}]]}
I20250110 02:51:30.411103 1184 consensus_queue.cc:260] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.414764 1184 ts_tablet_manager.cc:1428] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:30.415490 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:30.416107 1184 tablet_bootstrap.cc:492] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.425908 1184 tablet_bootstrap.cc:654] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.428794 2357 ts_tablet_manager.cc:1918] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.429051 2357 log.cc:1198] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8e79175853474cbea951f5a5686bf630
I20250110 02:51:30.429405 2357 ts_tablet_manager.cc:1939] T 8e79175853474cbea951f5a5686bf630 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.429714 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 141) took 6167 ms. Trace:
I20250110 02:51:30.430421 2357 rpcz_store.cc:276] 0110 02:51:24.262654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.262692 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:30.340406 (+6077714us) ts_tablet_manager.cc:1145] Deleting tablet 8e79175853474cbea951f5a5686bf630
0110 02:51:30.340416 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.423653 (+ 83237us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.428710 (+ 5057us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.429702 (+ 992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6065067}
I20250110 02:51:30.430997 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8e79175853474cbea951f5a5686bf630 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:30.431353 2357 tablet_replica.cc:331] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.432118 2357 raft_consensus.cc:2238] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.435734 2357 raft_consensus.cc:2267] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.436347 2357 ts_tablet_manager.cc:1905] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:30.438535 853 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 90940.9us
I20250110 02:51:30.439355 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.438593 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7fbdab52014dddbf63cfef69b09b7b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.440655 2345 raft_consensus.cc:491] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.440764 2345 raft_consensus.cc:513] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.441296 2345 leader_election.cc:290] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.442175 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.442211 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.333827 2908 consensus_queue.cc:237] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.446517 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e015a41a03f84c91a45e46e4b7b5e7a9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.446666 911 raft_consensus.cc:2463] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.453547 2357 ts_tablet_manager.cc:1918] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.453652 2357 log.cc:1198] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a81c031451104d5a93a1672b4b1918c5
I20250110 02:51:30.454049 2357 ts_tablet_manager.cc:1939] T a81c031451104d5a93a1672b4b1918c5 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.454406 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 142) took 6191 ms. Trace:
I20250110 02:51:30.454663 2357 rpcz_store.cc:276] 0110 02:51:24.262889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.262929 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:30.431309 (+6168380us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:30.431319 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.442906 (+ 11587us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.453370 (+ 10464us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.454390 (+ 1020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6155932}
I20250110 02:51:30.455109 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a81c031451104d5a93a1672b4b1918c5 (table default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]) successfully deleted
I20250110 02:51:30.455461 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 143) took 6192 ms. Trace:
I20250110 02:51:30.455528 2357 rpcz_store.cc:276] 0110 02:51:24.263431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.263707 (+ 276us) service_pool.cc:225] Handling call
0110 02:51:30.455374 (+6191667us) ts_tablet_manager.cc:1145] Deleting tablet 3510e86314144a2ca012aacf3d78eacc
0110 02:51:30.455384 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.455444 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6179953}
W20250110 02:51:30.455912 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because the tablet was not found. No further retry: Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:30.456158 2357 tablet_replica.cc:331] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.456421 2357 raft_consensus.cc:2238] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.456542 2357 raft_consensus.cc:2267] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.456998 2357 ts_tablet_manager.cc:1905] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.462041 2357 ts_tablet_manager.cc:1918] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.418665 1181 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.470427 1181 ts_tablet_manager.cc:1397] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.003s
I20250110 02:51:30.471067 1181 raft_consensus.cc:357] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.471223 1181 raft_consensus.cc:383] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.471262 1181 raft_consensus.cc:738] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.465118 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.471489 764 raft_consensus.cc:2388] T e03d53860ab744968693cf96467c3a19 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:30.471419 1181 consensus_queue.cc:260] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.466436 1184 tablet_bootstrap.cc:492] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.471871 1181 ts_tablet_manager.cc:1428] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:30.471980 1184 ts_tablet_manager.cc:1397] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.006s
I20250110 02:51:30.472126 1181 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.472606 1184 raft_consensus.cc:357] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.472765 1184 raft_consensus.cc:383] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.472810 1184 raft_consensus.cc:738] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.472985 1184 consensus_queue.cc:260] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.473378 1181 tablet_bootstrap.cc:654] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.426736 996 leader_election.cc:304] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.435374 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:30.435467 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.474200 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 47493.8us
I20250110 02:51:30.456019 2898 raft_consensus.cc:491] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.474488 2898 raft_consensus.cc:513] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.474922 2738 raft_consensus.cc:2798] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.475021 2738 raft_consensus.cc:695] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.475206 2738 consensus_queue.cc:237] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.475585 2898 leader_election.cc:290] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.475787 1184 ts_tablet_manager.cc:1428] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:30.364859 2905 leader_election.cc:290] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.475960 2357 log.cc:1198] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d00698a0a9ee4e5da69a745f4a3aa8b4
I20250110 02:51:30.476374 2357 ts_tablet_manager.cc:1939] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.476879 996 leader_election.cc:304] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.477653 996 leader_election.cc:304] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.477845 2898 raft_consensus.cc:2798] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.478413 2898 raft_consensus.cc:695] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.478766 2735 raft_consensus.cc:2798] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.478662 2898 consensus_queue.cc:237] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.478899 2735 raft_consensus.cc:695] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.479090 2735 consensus_queue.cc:237] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.480870 1184 tablet_bootstrap.cc:492] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.482052 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb595013e809480a814cb9c7aed856e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.482795 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 144) took 6218 ms. Trace:
I20250110 02:51:30.483096 2357 rpcz_store.cc:276] 0110 02:51:24.263990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264082 (+ 92us) service_pool.cc:225] Handling call
0110 02:51:30.456120 (+6192038us) ts_tablet_manager.cc:1145] Deleting tablet d00698a0a9ee4e5da69a745f4a3aa8b4
0110 02:51:30.456127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.459655 (+ 3528us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.461981 (+ 2326us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.482777 (+ 20796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6180652}
I20250110 02:51:30.483855 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]) successfully deleted
I20250110 02:51:30.484299 2357 tablet_replica.cc:331] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.484773 2357 raft_consensus.cc:2238] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.485204 2357 raft_consensus.cc:2267] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.485982 2357 ts_tablet_manager.cc:1905] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.486043 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cfc508c1b84299a2f16d307bb25354" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.490314 2906 raft_consensus.cc:491] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.490454 2906 raft_consensus.cc:513] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.491071 2906 leader_election.cc:290] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:30.494432 2357 ts_tablet_manager.cc:1918] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.494572 2357 log.cc:1198] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d424b4da66184bd19eccefad80621516
I20250110 02:51:30.494982 2357 ts_tablet_manager.cc:1939] T d424b4da66184bd19eccefad80621516 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.495467 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.495558 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 145) took 6231 ms. Trace:
I20250110 02:51:30.495625 768 raft_consensus.cc:2463] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:30.496289 711 leader_election.cc:343] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.495823 2357 rpcz_store.cc:276] 0110 02:51:24.264461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.264536 (+ 75us) service_pool.cc:225] Handling call
0110 02:51:30.484248 (+6219712us) ts_tablet_manager.cc:1145] Deleting tablet d424b4da66184bd19eccefad80621516
0110 02:51:30.484260 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.489574 (+ 5314us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.494317 (+ 4743us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.495547 (+ 1230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6208738}
I20250110 02:51:30.496788 996 leader_election.cc:304] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.496789 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet d424b4da66184bd19eccefad80621516 (table default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]) successfully deleted
W20250110 02:51:30.497009 996 leader_election.cc:343] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:30.497324 2904 raft_consensus.cc:2798] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.497408 2904 raft_consensus.cc:491] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.497464 2904 raft_consensus.cc:3054] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.499823 2904 raft_consensus.cc:513] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.501124 2904 leader_election.cc:290] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.501577 2357 tablet_replica.cc:331] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.501844 2357 raft_consensus.cc:2238] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.502097 2357 raft_consensus.cc:2267] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.502810 2357 ts_tablet_manager.cc:1905] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.504637 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.504942 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b90cc15dbf47d4b4dbaf12d588abb0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.505741 2352 ts_tablet_manager.cc:1918] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.505890 2352 log.cc:1198] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:30.506219 2352 ts_tablet_manager.cc:1939] T cdb7d0b9df3d48f0bb10eb6bc9b2cd6b P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.507053 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 177) took 6234 ms. Trace:
I20250110 02:51:30.507136 2352 rpcz_store.cc:276] 0110 02:51:24.272359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.272398 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:30.315057 (+6042659us) ts_tablet_manager.cc:1145] Deleting tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
0110 02:51:30.315074 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.454689 (+139615us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.505653 (+ 50964us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.507037 (+ 1384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6037872}
I20250110 02:51:30.507467 2352 tablet_replica.cc:331] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.507915 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.508977 994 leader_election.cc:343] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.509300 994 leader_election.cc:343] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.509495 994 leader_election.cc:343] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.509759 994 leader_election.cc:304] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:30.510040 994 leader_election.cc:343] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.510315 994 leader_election.cc:343] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.510564 994 leader_election.cc:343] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.510955 994 leader_election.cc:343] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.512012 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:30.512465 2906 raft_consensus.cc:2798] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.512526 2906 raft_consensus.cc:491] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.512593 2906 raft_consensus.cc:3054] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.514910 2906 raft_consensus.cc:513] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.515486 2906 leader_election.cc:290] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.516626 2549 raft_consensus.cc:491] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.516757 2549 raft_consensus.cc:513] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.517205 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b010e9c549214937889fa281542e752e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.517258 2549 leader_election.cc:290] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.523963 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.524119 774 raft_consensus.cc:3054] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:30.525283 996 leader_election.cc:343] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:30.526922 774 raft_consensus.cc:2463] T b25bbf8aa71e478587093627c715310f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.530526 996 leader_election.cc:304] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.530972 2899 raft_consensus.cc:2798] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.531140 2899 raft_consensus.cc:695] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.531471 2899 consensus_queue.cc:237] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.535192 2224 rpcz_store.cc:276] 0110 02:51:24.963617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.965913 (+ 2296us) service_pool.cc:225] Handling call
0110 02:51:30.370057 (+5404144us) ts_tablet_manager.cc:1145] Deleting tablet 1edb4e06e40c43438882e1eb622d788e
0110 02:51:30.370068 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.392204 (+ 22136us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.403042 (+ 10838us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.406360 (+ 3318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5403946}
I20250110 02:51:30.535718 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 211) took 5535 ms. Trace:
I20250110 02:51:30.535826 2224 rpcz_store.cc:276] 0110 02:51:25.000506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.000602 (+ 96us) service_pool.cc:225] Handling call
0110 02:51:30.535575 (+5534973us) ts_tablet_manager.cc:1145] Deleting tablet 24034fb3da6e47978cdc9b02e714e30d
0110 02:51:30.535588 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.535703 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5526050}
I20250110 02:51:30.536244 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 212) took 5526 ms. Trace:
I20250110 02:51:30.536334 2224 rpcz_store.cc:276] 0110 02:51:25.009894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.009935 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:30.536122 (+5526187us) ts_tablet_manager.cc:1145] Deleting tablet 818dcbf83303412d97b097913bcd36c3
0110 02:51:30.536142 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.536230 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5525043}
I20250110 02:51:30.536644 2224 tablet_replica.cc:331] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.536880 2224 raft_consensus.cc:2238] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.537055 2224 raft_consensus.cc:2267] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.537760 2224 ts_tablet_manager.cc:1905] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.540299 1184 tablet_bootstrap.cc:654] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.549641 2352 raft_consensus.cc:2238] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.549819 2352 raft_consensus.cc:2267] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.550311 2352 ts_tablet_manager.cc:1905] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.567274 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d53860ab744968693cf96467c3a19" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.568747 1433 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b5f162de87044005bdafe93292ff71b8 consensus state has no leader
I20250110 02:51:30.569164 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]) successfully deleted
W20250110 02:51:30.569633 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 24034fb3da6e47978cdc9b02e714e30d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24034fb3da6e47978cdc9b02e714e30d already in progress: creating tablet
W20250110 02:51:30.569993 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 818dcbf83303412d97b097913bcd36c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818dcbf83303412d97b097913bcd36c3 already in progress: creating tablet
W20250110 02:51:30.570816 709 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.571255 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 6e4d3270e3a04cc99f90943b9573ec7e (table default.Table_ecf7ad441f824cbe9e7384fb02a46754 [id=b52b8ac71efa4c5db265049fb25de5a7]) successfully deleted
I20250110 02:51:30.571447 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b (table default.Table_18721d8cd9034ade8df5d1035133a6a1 [id=0afabf48cd2241d3b284d3f7de80e6c8]) successfully deleted
I20250110 02:51:30.572938 1188 tablet_bootstrap.cc:492] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.573076 1188 ts_tablet_manager.cc:1397] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.279s user 0.005s sys 0.001s
I20250110 02:51:30.573787 1188 raft_consensus.cc:357] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.573994 1188 raft_consensus.cc:383] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.574046 1188 raft_consensus.cc:738] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:30.574244 853 consensus_peers.cc:487] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.574209 1188 consensus_queue.cc:260] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.574662 1188 ts_tablet_manager.cc:1428] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:30.574909 1188 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:30.574973 853 consensus_peers.cc:487] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:30.576071 853 consensus_peers.cc:487] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.576262 853 leader_election.cc:304] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.576777 2904 raft_consensus.cc:491] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.576901 2904 raft_consensus.cc:513] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.577402 2904 leader_election.cc:290] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.577953 2345 raft_consensus.cc:3054] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.580899 2345 raft_consensus.cc:2743] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.583576 2905 consensus_queue.cc:1035] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.584218 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.584475 770 raft_consensus.cc:2463] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.584784 996 leader_election.cc:304] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.585233 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d4f03a8c66446ea7c6d178cc7ef80a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.585635 1181 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.585758 1181 ts_tablet_manager.cc:1397] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.005s
I20250110 02:51:30.585243 2905 raft_consensus.cc:2798] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.586107 2905 raft_consensus.cc:491] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.586212 2905 raft_consensus.cc:3054] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.586560 1181 raft_consensus.cc:357] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.586743 1181 raft_consensus.cc:383] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.586815 1181 raft_consensus.cc:738] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.587031 1181 consensus_queue.cc:260] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.587486 1181 ts_tablet_manager.cc:1428] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250110 02:51:30.587677 853 consensus_peers.cc:487] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.587781 1181 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.589063 1181 tablet_bootstrap.cc:654] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.589850 2352 ts_tablet_manager.cc:1918] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.589954 2352 log.cc:1198] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1fb5412b78104d8085878c58b191ab27
W20250110 02:51:30.589970 711 leader_election.cc:343] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.590312 2352 ts_tablet_manager.cc:1939] T 1fb5412b78104d8085878c58b191ab27 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.591801 2345 raft_consensus.cc:491] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8899dd1aeb384e4ea4523f5fd225c888)
I20250110 02:51:30.591926 2345 raft_consensus.cc:513] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.592412 2345 leader_election.cc:290] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.593201 853 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907
W20250110 02:51:30.597036 853 leader_election.cc:343] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:30.597175 853 leader_election.cc:304] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.597417 2933 raft_consensus.cc:2743] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 02:51:30.598809 2691 raft_consensus.cc:491] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.598944 2691 raft_consensus.cc:513] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.599460 2691 leader_election.cc:290] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.601186 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.604662 774 raft_consensus.cc:2388] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:30.607159 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:30.607666 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.608014 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d4f03a8c66446ea7c6d178cc7ef80a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.608103 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.608019 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba0cb7970f1453dac256b255ed7cac0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.609607 709 leader_election.cc:343] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.609719 709 leader_election.cc:304] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.610059 2549 raft_consensus.cc:2743] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:30.610395 709 leader_election.cc:343] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.610488 709 leader_election.cc:304] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.610811 2549 raft_consensus.cc:2743] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:30.611094 709 leader_election.cc:343] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.611188 709 leader_election.cc:304] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.611487 2549 raft_consensus.cc:2743] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:30.611788 709 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.613127 709 consensus_peers.cc:487] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.613996 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 178) took 6341 ms. Trace:
I20250110 02:51:30.614073 2352 rpcz_store.cc:276] 0110 02:51:24.272714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.272757 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:30.507414 (+6234657us) ts_tablet_manager.cc:1145] Deleting tablet 1fb5412b78104d8085878c58b191ab27
0110 02:51:30.507425 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.553206 (+ 45781us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.589743 (+ 36537us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.613980 (+ 24237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41937,"tablet-delete.queue_time_us":6234519}
I20250110 02:51:30.614354 2352 tablet_replica.cc:331] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.614562 2352 raft_consensus.cc:2238] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.614694 2352 raft_consensus.cc:2267] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.615240 2352 ts_tablet_manager.cc:1905] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.615288 2357 ts_tablet_manager.cc:1918] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.615379 2357 log.cc:1198] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7e554d00d05b4fa7a14c9b6159f9d726
I20250110 02:51:30.615660 2357 ts_tablet_manager.cc:1939] T 7e554d00d05b4fa7a14c9b6159f9d726 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.616461 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 146) took 6333 ms. Trace:
I20250110 02:51:30.616524 2357 rpcz_store.cc:276] 0110 02:51:24.283371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.283695 (+ 324us) service_pool.cc:225] Handling call
0110 02:51:30.501522 (+6217827us) ts_tablet_manager.cc:1145] Deleting tablet 7e554d00d05b4fa7a14c9b6159f9d726
0110 02:51:30.501534 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.528156 (+ 26622us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.615191 (+ 87035us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.616444 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6217665}
I20250110 02:51:30.616935 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]) successfully deleted
I20250110 02:51:30.617233 2357 tablet_replica.cc:331] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.617413 2357 raft_consensus.cc:2238] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.617533 2357 raft_consensus.cc:2267] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.617993 2357 ts_tablet_manager.cc:1905] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.618537 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1fb5412b78104d8085878c58b191ab27 (table default.Table_4b2b1d61b1f24f2cb7a8d78f176f37b0 [id=b335adce4a3142d0a189d5bb71199a57]) successfully deleted
I20250110 02:51:30.636523 1184 tablet_bootstrap.cc:492] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.636703 1184 ts_tablet_manager.cc:1397] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.009s
I20250110 02:51:30.637379 1184 raft_consensus.cc:357] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.637586 1184 raft_consensus.cc:383] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.637671 1184 raft_consensus.cc:738] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.637924 1184 consensus_queue.cc:260] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.638599 1184 ts_tablet_manager.cc:1428] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:30.638953 1184 tablet_bootstrap.cc:492] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.640599 1184 tablet_bootstrap.cc:654] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.644551 1188 tablet_bootstrap.cc:654] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.645586 2352 ts_tablet_manager.cc:1918] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.589196 2905 raft_consensus.cc:513] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.646248 2905 leader_election.cc:290] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.594594 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 83437.8us
I20250110 02:51:30.600314 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7564208da99449cd97338a41b263e750" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.646898 994 leader_election.cc:343] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.646998 994 leader_election.cc:304] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:30.647891 994 leader_election.cc:343] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.648340 994 leader_election.cc:343] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.648721 2904 consensus_queue.cc:1035] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.648372 2289 raft_consensus.cc:2743] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.655817 2549 raft_consensus.cc:491] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.655937 2549 raft_consensus.cc:513] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.656471 2549 leader_election.cc:290] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.656484 2352 log.cc:1198] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:30.649791 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e015a41a03f84c91a45e46e4b7b5e7a9" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.656759 915 raft_consensus.cc:3054] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.656864 2352 ts_tablet_manager.cc:1939] T 65ae1c19d2dc4778a1cebddbd505f907 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.657497 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 179) took 6384 ms. Trace:
I20250110 02:51:30.657596 2352 rpcz_store.cc:276] 0110 02:51:24.273076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.273113 (+ 37us) service_pool.cc:225] Handling call
0110 02:51:30.614318 (+6341205us) ts_tablet_manager.cc:1145] Deleting tablet 65ae1c19d2dc4778a1cebddbd505f907
0110 02:51:30.614326 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.642654 (+ 28328us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.645507 (+ 2853us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.657482 (+ 11975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6341063}
I20250110 02:51:30.657936 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 180) took 6384 ms. Trace:
I20250110 02:51:30.659999 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 65ae1c19d2dc4778a1cebddbd505f907 (table default.Table_61492b773a714042b0532061e607c827 [id=61a005523555495e8c83ddefea86a214]) successfully deleted
I20250110 02:51:30.661260 2905 consensus_queue.cc:1035] T e03d53860ab744968693cf96467c3a19 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.650691 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.659523 915 raft_consensus.cc:2463] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.650283 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.661840 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:30.664441 709 leader_election.cc:343] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.664655 2456 raft_consensus.cc:491] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.664809 2456 raft_consensus.cc:513] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.665599 2456 leader_election.cc:290] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.666533 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.666735 764 raft_consensus.cc:2463] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.667263 996 leader_election.cc:304] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.667474 2898 raft_consensus.cc:2798] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.667563 2898 raft_consensus.cc:491] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.667657 2898 raft_consensus.cc:3054] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.670403 2352 rpcz_store.cc:276] 0110 02:51:24.273429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.273465 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:30.657845 (+6384380us) ts_tablet_manager.cc:1145] Deleting tablet a0ddab9e847445999ec97eac4352d9c2
0110 02:51:30.657853 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.657924 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6384242}
I20250110 02:51:30.670537 853 leader_election.cc:304] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.670743 2908 raft_consensus.cc:3054] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.670779 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 182) took 6370 ms. Trace:
I20250110 02:51:30.670879 2352 rpcz_store.cc:276] 0110 02:51:24.299989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.300036 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:30.670679 (+6370643us) ts_tablet_manager.cc:1145] Deleting tablet 3fb805b3e537466ab7fe0185421ad5aa
0110 02:51:30.670688 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.670766 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6370487}
W20250110 02:51:30.671633 994 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.671815 994 leader_election.cc:304] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.671434 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.672457 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069d19975e2a4bb0a23b4e9983f411d3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.672639 770 raft_consensus.cc:3054] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.672819 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 190) took 5749 ms. Trace:
W20250110 02:51:30.672279 996 consensus_peers.cc:487] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:30.673220 994 leader_election.cc:343] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.673326 2352 rpcz_store.cc:276] 0110 02:51:24.923580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.923627 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:30.672740 (+5749113us) ts_tablet_manager.cc:1145] Deleting tablet b45b91d084a44c1e8cdf1553967878e2
0110 02:51:30.672749 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.672807 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5748935}
I20250110 02:51:30.673599 2352 tablet_replica.cc:331] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:30.673806 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a0ddab9e847445999ec97eac4352d9c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0ddab9e847445999ec97eac4352d9c2 already in progress: creating tablet
I20250110 02:51:30.673044 2908 raft_consensus.cc:2743] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.674060 2352 raft_consensus.cc:2238] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:30.674075 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
W20250110 02:51:30.674300 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b45b91d084a44c1e8cdf1553967878e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b45b91d084a44c1e8cdf1553967878e2 already in progress: creating tablet
I20250110 02:51:30.673373 2738 raft_consensus.cc:2798] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.673460 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:30.675056 994 leader_election.cc:343] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.675321 2352 raft_consensus.cc:2267] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.675374 2738 raft_consensus.cc:695] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.675827 2352 ts_tablet_manager.cc:1905] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.635661 2600 raft_consensus.cc:513] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.676767 2738 consensus_queue.cc:237] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:30.681102 711 leader_election.cc:343] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.681191 711 leader_election.cc:304] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.681385 2549 raft_consensus.cc:2743] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.683877 2937 tablet.cc:1721] T 09fd9734714a4847ab2a4ebc37af8960 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:30.684521 2938 tablet.cc:1721] T 09fd9734714a4847ab2a4ebc37af8960 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:30.687841 770 raft_consensus.cc:2463] T 069d19975e2a4bb0a23b4e9983f411d3 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.688164 996 leader_election.cc:304] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.688578 2735 raft_consensus.cc:2798] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.688680 2735 raft_consensus.cc:695] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.688875 2735 consensus_queue.cc:237] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.696231 2352 ts_tablet_manager.cc:1918] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.696404 2352 log.cc:1198] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/1edb4e06e40c43438882e1eb622d788e
I20250110 02:51:30.696703 2600 leader_election.cc:290] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.697048 2352 ts_tablet_manager.cc:1939] T 1edb4e06e40c43438882e1eb622d788e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.697475 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 191) took 5734 ms. Trace:
I20250110 02:51:30.697659 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.697623 2352 rpcz_store.cc:276] 0110 02:51:24.962745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.962801 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:30.673566 (+5710765us) ts_tablet_manager.cc:1145] Deleting tablet 1edb4e06e40c43438882e1eb622d788e
0110 02:51:30.673573 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.680739 (+ 7166us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.696107 (+ 15368us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.697442 (+ 1335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5710573}
I20250110 02:51:30.697870 774 raft_consensus.cc:3054] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.698136 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]) successfully deleted
I20250110 02:51:30.698439 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 192) took 5715 ms. Trace:
I20250110 02:51:30.698618 2352 rpcz_store.cc:276] 0110 02:51:24.983146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.983203 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:30.698201 (+5714998us) ts_tablet_manager.cc:1145] Deleting tablet 24034fb3da6e47978cdc9b02e714e30d
0110 02:51:30.698212 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.698423 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5714801}
W20250110 02:51:30.699172 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 24034fb3da6e47978cdc9b02e714e30d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24034fb3da6e47978cdc9b02e714e30d already in progress: creating tablet
I20250110 02:51:30.699388 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 193) took 5714 ms. Trace:
I20250110 02:51:30.699518 2352 rpcz_store.cc:276] 0110 02:51:24.984848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.985128 (+ 280us) service_pool.cc:225] Handling call
0110 02:51:30.699230 (+5714102us) ts_tablet_manager.cc:1145] Deleting tablet 818dcbf83303412d97b097913bcd36c3
0110 02:51:30.699241 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.699372 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5713915}
W20250110 02:51:30.699998 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 818dcbf83303412d97b097913bcd36c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818dcbf83303412d97b097913bcd36c3 already in progress: creating tablet
I20250110 02:51:30.700474 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 199) took 5668 ms. Trace:
I20250110 02:51:30.700587 2352 rpcz_store.cc:276] 0110 02:51:25.032141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.032191 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:30.700036 (+5667845us) ts_tablet_manager.cc:1145] Deleting tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
0110 02:51:30.700043 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.700132 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5667682}
W20250110 02:51:30.700984 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a already in progress: creating tablet
I20250110 02:51:30.701144 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 200) took 5310 ms. Trace:
I20250110 02:51:30.701256 2352 rpcz_store.cc:276] 0110 02:51:25.391080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.392710 (+ 1630us) service_pool.cc:225] Handling call
0110 02:51:30.701006 (+5308296us) ts_tablet_manager.cc:1145] Deleting tablet fe860d60b3194dc5a2e7a94d39d4b8dd
0110 02:51:30.701015 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.701129 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5308103}
W20250110 02:51:30.701632 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
I20250110 02:51:30.701732 2352 tablet_replica.cc:331] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.701931 2898 raft_consensus.cc:513] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.702024 2352 raft_consensus.cc:2238] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.702235 2352 raft_consensus.cc:2267] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.702605 2898 leader_election.cc:290] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.703197 2352 ts_tablet_manager.cc:1905] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.703366 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.703570 770 raft_consensus.cc:3054] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.706960 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761ef0d1e5542b1979c29bb2d579489" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.707126 912 raft_consensus.cc:3054] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.707172 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.709739 912 raft_consensus.cc:2463] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.711973 2352 ts_tablet_manager.cc:1918] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.712127 2352 log.cc:1198] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/6555b0a4491046bf8f5cd4ee406c3955
W20250110 02:51:30.712308 994 leader_election.cc:343] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.712505 994 leader_election.cc:304] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters:
I20250110 02:51:30.713208 2352 ts_tablet_manager.cc:1939] T 6555b0a4491046bf8f5cd4ee406c3955 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.713653 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 201) took 5301 ms. Trace:
I20250110 02:51:30.713734 2600 raft_consensus.cc:2798] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.713872 2600 raft_consensus.cc:695] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.713793 2352 rpcz_store.cc:276] 0110 02:51:25.411964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.412016 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:30.701674 (+5289658us) ts_tablet_manager.cc:1145] Deleting tablet 6555b0a4491046bf8f5cd4ee406c3955
0110 02:51:30.701703 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.707406 (+ 5703us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.711869 (+ 4463us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.713612 (+ 1743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5287072}
I20250110 02:51:30.714092 2600 consensus_queue.cc:237] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.714437 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]) successfully deleted
I20250110 02:51:30.714573 2352 tablet_replica.cc:331] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.714843 2352 raft_consensus.cc:2238] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.715066 2352 raft_consensus.cc:2267] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.715787 2352 ts_tablet_manager.cc:1905] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.717973 774 raft_consensus.cc:2463] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.718621 770 raft_consensus.cc:2463] T ae3033595f824affb2d300c3964e1e8e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.722503 996 leader_election.cc:304] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:30.722853 2738 raft_consensus.cc:2798] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.722954 2738 raft_consensus.cc:695] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.723141 2738 consensus_queue.cc:237] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.726429 2352 ts_tablet_manager.cc:1918] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.726590 2352 log.cc:1198] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/940445840db3441598eba57d3ccb8c9c
I20250110 02:51:30.727106 2352 ts_tablet_manager.cc:1939] T 940445840db3441598eba57d3ccb8c9c P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.727568 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 203) took 4416 ms. Trace:
I20250110 02:51:30.727736 2352 rpcz_store.cc:276] 0110 02:51:26.311277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.311387 (+ 110us) service_pool.cc:225] Handling call
0110 02:51:30.714489 (+4403102us) ts_tablet_manager.cc:1145] Deleting tablet 940445840db3441598eba57d3ccb8c9c
0110 02:51:30.714498 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.721327 (+ 6829us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.726309 (+ 4982us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.727552 (+ 1243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4402918}
I20250110 02:51:30.728228 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 940445840db3441598eba57d3ccb8c9c (table default.Table_9f5ae5046a4b45e5a5f2cace5ca40e46 [id=1e73398ddfd8490ba912f3ace7bd5415]) successfully deleted
I20250110 02:51:30.728399 2352 tablet_replica.cc:331] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.728741 2352 raft_consensus.cc:2238] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.729013 2352 raft_consensus.cc:2267] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.729874 2224 ts_tablet_manager.cc:1918] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.729902 2352 ts_tablet_manager.cc:1905] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.730022 2224 log.cc:1198] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/65285bce60fd4015b085deb7442eea6e
I20250110 02:51:30.730356 2224 ts_tablet_manager.cc:1939] T 65285bce60fd4015b085deb7442eea6e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.730691 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 215) took 5715 ms. Trace:
I20250110 02:51:30.730791 2224 rpcz_store.cc:276] 0110 02:51:25.015561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.015599 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:30.536593 (+5520994us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:30.536600 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.624055 (+ 87455us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.729782 (+105727us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.730677 (+ 895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5520845}
I20250110 02:51:30.731181 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 65285bce60fd4015b085deb7442eea6e (table default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]) successfully deleted
I20250110 02:51:30.734969 2947 tablet.cc:1721] T 09fd9734714a4847ab2a4ebc37af8960 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:30.738324 1184 tablet_bootstrap.cc:492] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.738474 1181 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.738528 1184 ts_tablet_manager.cc:1397] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.001s
I20250110 02:51:30.738708 1181 ts_tablet_manager.cc:1397] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.001s
I20250110 02:51:30.739346 1184 raft_consensus.cc:357] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.739571 1184 raft_consensus.cc:383] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.739655 1184 raft_consensus.cc:738] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.739722 1181 raft_consensus.cc:357] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.740000 1181 raft_consensus.cc:383] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.739946 1184 consensus_queue.cc:260] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.740083 1181 raft_consensus.cc:738] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.740334 1181 consensus_queue.cc:260] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.740938 1181 ts_tablet_manager.cc:1428] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:30.741261 1181 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.743137 1181 tablet_bootstrap.cc:654] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.743376 1184 ts_tablet_manager.cc:1428] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250110 02:51:30.743624 1184 tablet_bootstrap.cc:492] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.745028 1184 tablet_bootstrap.cc:654] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.752126 2352 ts_tablet_manager.cc:1918] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.752223 2352 log.cc:1198] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:30.766449 2352 ts_tablet_manager.cc:1939] T e015a41a03f84c91a45e46e4b7b5e7a9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.766753 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 204) took 4455 ms. Trace:
I20250110 02:51:30.766829 2352 rpcz_store.cc:276] 0110 02:51:26.311357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.311404 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:30.728284 (+4416880us) ts_tablet_manager.cc:1145] Deleting tablet e015a41a03f84c91a45e46e4b7b5e7a9
0110 02:51:30.728292 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.734162 (+ 5870us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.752047 (+ 17885us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.766738 (+ 14691us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32,"tablet-delete.queue_time_us":4416715}
I20250110 02:51:30.767170 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 205) took 4452 ms. Trace:
I20250110 02:51:30.767227 2352 rpcz_store.cc:276] 0110 02:51:26.314518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.314564 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:30.767084 (+4452520us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:30.767092 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.767158 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4452361}
I20250110 02:51:30.767419 2352 tablet_replica.cc:331] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.767601 2352 raft_consensus.cc:2238] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.767867 2735 consensus_queue.cc:1035] T 8bc32061409246a2b280dde4f9180a8e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.768428 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e015a41a03f84c91a45e46e4b7b5e7a9 (table default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]) successfully deleted
I20250110 02:51:30.768548 1188 tablet_bootstrap.cc:492] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:30.768687 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because the tablet was not found. No further retry: Not found: Tablet not found: 0d5ddbc6b02742609bc2ca84c863a9d0
I20250110 02:51:30.768774 1188 ts_tablet_manager.cc:1397] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.194s user 0.003s sys 0.001s
I20250110 02:51:30.772212 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 216) took 5755 ms. Trace:
I20250110 02:51:30.772313 2224 rpcz_store.cc:276] 0110 02:51:25.016637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.016679 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:30.772098 (+5755419us) ts_tablet_manager.cc:1145] Deleting tablet b4f1782c46a44ed091f4c533226c71f2
0110 02:51:30.772107 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.772196 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5755261}
W20250110 02:51:30.772683 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:30.776821 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 218) took 5759 ms. Trace:
I20250110 02:51:30.776937 2224 rpcz_store.cc:276] 0110 02:51:25.016841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.016876 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:30.776726 (+5759850us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:30.776738 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.776803 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5759714}
I20250110 02:51:30.777268 2224 tablet_replica.cc:331] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.777587 2224 raft_consensus.cc:2238] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.777786 2224 raft_consensus.cc:2267] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.778759 2504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=646 overwritten=0 applied=646 ignored=0} inserts{seen=609 ignored=0} mutations{seen=741 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 02:51:30.779398 2504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: Bootstrap complete.
W20250110 02:51:30.779758 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because the tablet was not found. No further retry: Not found: Tablet not found: b375a932156d4abb908dbf327689b0f9
W20250110 02:51:30.781067 853 consensus_peers.cc:487] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d424b4da66184bd19eccefad80621516. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.782661 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0e95ffbc049a39e30d6670f35382c (DEFAULT_TABLE table=default.Table_e634410b037147489f005fd9486c1a93 [id=221b6efd55d441549f97fc1d818615f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.782996 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b0e95ffbc049a39e30d6670f35382c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.789721 2224 ts_tablet_manager.cc:1905] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.796670 2504 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:30.798570 2345 raft_consensus.cc:491] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.798699 2345 raft_consensus.cc:513] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.799242 2345 leader_election.cc:290] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.800401 853 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:30.800402 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a3668b902f6a4ed8adc606e1d6d08225 (DEFAULT_TABLE table=default.Table_77210dde99b74eb8b00f98f106690a9f [id=c614c5c177e34967a0ef26eb5eb5f4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.800717 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3668b902f6a4ed8adc606e1d6d08225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.801440 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00e97e496b54dcf82b7182591d6f4aa" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.801635 1061 raft_consensus.cc:2463] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 0.
I20250110 02:51:30.801918 853 leader_election.cc:304] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.802546 895 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0e95ffbc049a39e30d6670f35382c (DEFAULT_TABLE table=default.Table_e634410b037147489f005fd9486c1a93 [id=221b6efd55d441549f97fc1d818615f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.802832 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b0e95ffbc049a39e30d6670f35382c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.804513 742 tablet_service.cc:1467] Processing CreateTablet for tablet a3668b902f6a4ed8adc606e1d6d08225 (DEFAULT_TABLE table=default.Table_77210dde99b74eb8b00f98f106690a9f [id=c614c5c177e34967a0ef26eb5eb5f4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.804836 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3668b902f6a4ed8adc606e1d6d08225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.768465 2199 raft_consensus.cc:491] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.806468 2199 raft_consensus.cc:513] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.806977 2199 leader_election.cc:290] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.785122 2549 consensus_queue.cc:1035] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.808125 2345 raft_consensus.cc:2798] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.808193 2345 raft_consensus.cc:491] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.808246 2345 raft_consensus.cc:3054] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.808282 894 tablet_service.cc:1467] Processing CreateTablet for tablet a3668b902f6a4ed8adc606e1d6d08225 (DEFAULT_TABLE table=default.Table_77210dde99b74eb8b00f98f106690a9f [id=c614c5c177e34967a0ef26eb5eb5f4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.808607 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3668b902f6a4ed8adc606e1d6d08225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.810680 2345 raft_consensus.cc:513] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.812072 2345 leader_election.cc:290] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Requested vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:30.815467 853 leader_election.cc:343] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:30.815732 2352 raft_consensus.cc:2267] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.815757 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071bbce1706c43b6bb690c7de1ead3fe" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.815899 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00e97e496b54dcf82b7182591d6f4aa" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:30.815850 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071bbce1706c43b6bb690c7de1ead3fe" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.816028 1061 raft_consensus.cc:3054] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.816280 2352 ts_tablet_manager.cc:1905] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:30.816365 709 leader_election.cc:343] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.817759 711 leader_election.cc:343] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.817855 711 leader_election.cc:304] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.818759 2504 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc934cc708045e9858ee8f8e64660cd, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.819716 2504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 646, Last appended: 3.646, Last appended by leader: 646, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:30.820477 1188 raft_consensus.cc:357] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.821511 1188 raft_consensus.cc:383] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.822180 1188 raft_consensus.cc:738] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.823519 2352 ts_tablet_manager.cc:1918] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.823596 2352 log.cc:1198] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:30.823096 1188 consensus_queue.cc:260] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.823872 2352 ts_tablet_manager.cc:1939] T 6e0b2ec1d5814144bc58261e6c6e8d35 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.825642 1188 ts_tablet_manager.cc:1428] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20250110 02:51:30.826704 2456 raft_consensus.cc:491] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.826812 2456 raft_consensus.cc:513] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.827343 2456 leader_election.cc:290] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.827797 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:30.828444 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.828724 2929 raft_consensus.cc:491] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.828842 2929 raft_consensus.cc:513] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.829288 895 tablet_service.cc:1467] Processing CreateTablet for tablet 0b9b579176164c7a8aeeab632d764670 (DEFAULT_TABLE table=default.Table_1646191c9c32465fad0850bda2dd3367 [id=1e3734d0995e4bbb825ae3fcc7300f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.829290 742 tablet_service.cc:1467] Processing CreateTablet for tablet 0b9b579176164c7a8aeeab632d764670 (DEFAULT_TABLE table=default.Table_1646191c9c32465fad0850bda2dd3367 [id=1e3734d0995e4bbb825ae3fcc7300f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.829427 2929 leader_election.cc:290] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.829532 770 raft_consensus.cc:2463] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.829669 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b9b579176164c7a8aeeab632d764670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.799521 2357 ts_tablet_manager.cc:1918] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:30.800873 756 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0e95ffbc049a39e30d6670f35382c (DEFAULT_TABLE table=default.Table_e634410b037147489f005fd9486c1a93 [id=221b6efd55d441549f97fc1d818615f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.830003 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b0e95ffbc049a39e30d6670f35382c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.830212 996 leader_election.cc:304] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.830561 2929 raft_consensus.cc:2798] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.830624 2929 raft_consensus.cc:491] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.830673 2929 raft_consensus.cc:3054] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.832921 2929 raft_consensus.cc:513] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.833475 2929 leader_election.cc:290] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.833961 2549 raft_consensus.cc:2743] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.833994 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.834111 2549 raft_consensus.cc:491] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:30.834156 764 raft_consensus.cc:3054] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.834201 2549 raft_consensus.cc:513] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.834692 2549 leader_election.cc:290] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:30.835331 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.835757 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.835915 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:30.836138 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:30.836158 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:30.836261 764 raft_consensus.cc:2463] T 718d3add9d0a4e4b8633f2c977e316e6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.836300 770 raft_consensus.cc:2463] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:30.836686 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0b9b579176164c7a8aeeab632d764670 (DEFAULT_TABLE table=default.Table_1646191c9c32465fad0850bda2dd3367 [id=1e3734d0995e4bbb825ae3fcc7300f96]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:30.836783 709 leader_election.cc:343] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.837337 996 leader_election.cc:304] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.837355 2357 log.cc:1198] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7564208da99449cd97338a41b263e750
I20250110 02:51:30.837584 996 leader_election.cc:304] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:30.837807 2456 consensus_queue.cc:1035] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.838156 2456 raft_consensus.cc:2798] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.838256 2456 raft_consensus.cc:695] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.838291 2738 raft_consensus.cc:2798] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:30.838342 2738 raft_consensus.cc:491] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:30.838428 2738 raft_consensus.cc:3054] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.838503 2456 consensus_queue.cc:237] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.839268 2357 ts_tablet_manager.cc:1939] T 7564208da99449cd97338a41b263e750 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:30.839979 994 leader_election.cc:343] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.840240 994 leader_election.cc:343] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:30.840431 994 leader_election.cc:343] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.840778 2738 raft_consensus.cc:513] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.841074 1061 raft_consensus.cc:2463] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f9597077934f1a9814d284ed06d13d in term 1.
I20250110 02:51:30.841871 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:30.842300 994 leader_election.cc:343] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.842610 2738 leader_election.cc:290] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:30.842818 1181 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:30.843055 1181 ts_tablet_manager.cc:1397] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.102s user 0.010s sys 0.012s
I20250110 02:51:30.843024 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:30.843207 764 raft_consensus.cc:3054] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:30.844027 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b9b579176164c7a8aeeab632d764670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.844453 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 206) took 4529 ms. Trace:
I20250110 02:51:30.844580 2352 rpcz_store.cc:276] 0110 02:51:26.314622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.314755 (+ 133us) service_pool.cc:225] Handling call
0110 02:51:30.767386 (+4452631us) ts_tablet_manager.cc:1145] Deleting tablet 6e0b2ec1d5814144bc58261e6c6e8d35
0110 02:51:30.767393 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.820642 (+ 53249us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.823452 (+ 2810us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.844436 (+ 20984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4452521}
I20250110 02:51:30.845046 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 6e0b2ec1d5814144bc58261e6c6e8d35 (table default.Table_c2f177ce9d6a420bbc77b33929ccaa7a [id=f18e7a9e2e274c2d86fb5d9455ecc4fe]) successfully deleted
I20250110 02:51:30.845220 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 208) took 4530 ms. Trace:
I20250110 02:51:30.845639 2352 rpcz_store.cc:276] 0110 02:51:26.314783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.314889 (+ 106us) service_pool.cc:225] Handling call
0110 02:51:30.845063 (+4530174us) ts_tablet_manager.cc:1145] Deleting tablet 46a4745356e3423f9d4892d242da1c48
0110 02:51:30.845071 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.845206 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4530039}
W20250110 02:51:30.846624 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 46a4745356e3423f9d4892d242da1c48 because the tablet was not found. No further retry: Not found: Tablet not found: 46a4745356e3423f9d4892d242da1c48
I20250110 02:51:30.847242 2352 tablet_replica.cc:331] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.848008 2352 raft_consensus.cc:2238] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:30.849174 853 leader_election.cc:304] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888; no voters: 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:30.849464 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 86d96c45f70a468a91ea6f4694e7877f (DEFAULT_TABLE table=default.Table_627f1a96679c4cad8d88b188e4d769b8 [id=744db20489294f98ad2a37119678c2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.849647 2908 raft_consensus.cc:2798] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.849754 2908 raft_consensus.cc:695] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Becoming Leader. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Running, Role: LEADER
I20250110 02:51:30.849840 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:30.850015 2908 consensus_queue.cc:237] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.852385 1184 tablet_bootstrap.cc:492] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.850787 2352 raft_consensus.cc:2267] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.852552 1184 ts_tablet_manager.cc:1397] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.109s user 0.001s sys 0.003s
I20250110 02:51:30.853262 1184 raft_consensus.cc:357] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.853842 1181 raft_consensus.cc:357] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.854188 1181 raft_consensus.cc:383] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.854331 1181 raft_consensus.cc:738] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.856529 1188 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:30.856961 764 raft_consensus.cc:2463] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:30.858866 2352 ts_tablet_manager.cc:1905] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:30.867197 711 leader_election.cc:343] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:30.868975 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7c151d25b1714b2d86e83e73a58b7b49 (DEFAULT_TABLE table=default.Table_3fbaaf982d4c443a8d174437ef013ab2 [id=602370f8e86e4145a05c8b9978c25ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.869357 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c151d25b1714b2d86e83e73a58b7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.872702 1184 raft_consensus.cc:383] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.868975 894 tablet_service.cc:1467] Processing CreateTablet for tablet 8435d5dd16b146c98ae86293a074445f (DEFAULT_TABLE table=default.Table_9883b2a0f5e448df83b62a280a1cb915 [id=1fb43093bd654d8e94fb42cdfa0e565b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.874650 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8435d5dd16b146c98ae86293a074445f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.875051 711 leader_election.cc:304] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:30.875342 2199 raft_consensus.cc:2743] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:30.875766 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7c151d25b1714b2d86e83e73a58b7b49 (DEFAULT_TABLE table=default.Table_3fbaaf982d4c443a8d174437ef013ab2 [id=602370f8e86e4145a05c8b9978c25ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.875890 748 tablet_service.cc:1467] Processing CreateTablet for tablet c1885ab76b5c424d82fddd6b5481e4d6 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.876562 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8435d5dd16b146c98ae86293a074445f (DEFAULT_TABLE table=default.Table_9883b2a0f5e448df83b62a280a1cb915 [id=1fb43093bd654d8e94fb42cdfa0e565b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.876812 1184 raft_consensus.cc:738] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.877101 1184 consensus_queue.cc:260] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.877846 1184 ts_tablet_manager.cc:1428] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20250110 02:51:30.878942 883 tablet_service.cc:1467] Processing CreateTablet for tablet c1885ab76b5c424d82fddd6b5481e4d6 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.879233 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1885ab76b5c424d82fddd6b5481e4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.883849 881 tablet_service.cc:1467] Processing CreateTablet for tablet af8ff3f5bb2f4240837130df790fa6d8 (DEFAULT_TABLE table=default.Table_2433934279a049aca30f6cccd6c71952 [id=2ee5bd916ba84d42aa04b8991e381465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.884171 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8ff3f5bb2f4240837130df790fa6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.884186 753 tablet_service.cc:1467] Processing CreateTablet for tablet af8ff3f5bb2f4240837130df790fa6d8 (DEFAULT_TABLE table=default.Table_2433934279a049aca30f6cccd6c71952 [id=2ee5bd916ba84d42aa04b8991e381465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.889075 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28d8769f364e10aad54b60fb221f35 (DEFAULT_TABLE table=default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.890934 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28d8769f364e10aad54b60fb221f35 (DEFAULT_TABLE table=default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.891184 756 tablet_service.cc:1467] Processing CreateTablet for tablet 86d96c45f70a468a91ea6f4694e7877f (DEFAULT_TABLE table=default.Table_627f1a96679c4cad8d88b188e4d769b8 [id=744db20489294f98ad2a37119678c2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.891464 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e28d8769f364e10aad54b60fb221f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.891541 754 tablet_service.cc:1467] Processing CreateTablet for tablet 0e25b2b512e6405b956d49944c418c19 (DEFAULT_TABLE table=default.Table_06192f79203e4dd4bda8cda74e08e7c7 [id=8af0a73455e54ea5b8ba1872c3238b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.891863 2352 ts_tablet_manager.cc:1918] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.892343 2352 log.cc:1198] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:30.894177 894 tablet_service.cc:1467] Processing CreateTablet for tablet 0e25b2b512e6405b956d49944c418c19 (DEFAULT_TABLE table=default.Table_06192f79203e4dd4bda8cda74e08e7c7 [id=8af0a73455e54ea5b8ba1872c3238b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.894201 745 tablet_service.cc:1467] Processing CreateTablet for tablet bd669ba7a4bf47b39063f61d33556724 (DEFAULT_TABLE table=default.Table_2d3082a17dfd4f50b025f2da061be352 [id=4e490b91659d4526b1dbb0841112b55f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.894487 2352 ts_tablet_manager.cc:1939] T b382d634db9642df8e00c5ccaa5787a9 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.894999 889 tablet_service.cc:1467] Processing CreateTablet for tablet bd669ba7a4bf47b39063f61d33556724 (DEFAULT_TABLE table=default.Table_2d3082a17dfd4f50b025f2da061be352 [id=4e490b91659d4526b1dbb0841112b55f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.894510 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e25b2b512e6405b956d49944c418c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.895649 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 210) took 4580 ms. Trace:
I20250110 02:51:30.895965 2352 rpcz_store.cc:276] 0110 02:51:26.314930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315045 (+ 115us) service_pool.cc:225] Handling call
0110 02:51:30.847023 (+4531978us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:30.847047 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.880117 (+ 33070us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.891598 (+ 11481us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.895597 (+ 3999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531640}
I20250110 02:51:30.896842 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet b382d634db9642df8e00c5ccaa5787a9 (table default.Table_86a6941cdab84e57989003542ba3eb76 [id=e79344de353f46c7af10e47f7e4a8b84]) successfully deleted
I20250110 02:51:30.897171 2352 tablet_replica.cc:331] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.898420 2352 raft_consensus.cc:2238] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:30.898794 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd669ba7a4bf47b39063f61d33556724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.898706 739 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d7aed46234a9095f0013f8ab219c1 (DEFAULT_TABLE table=default.Table_acfdadc9edbf4eaf94f0a667273d93ec [id=1aae6180ec594da591e444f6cd4f04c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.899899 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d7aed46234a9095f0013f8ab219c1 (DEFAULT_TABLE table=default.Table_acfdadc9edbf4eaf94f0a667273d93ec [id=1aae6180ec594da591e444f6cd4f04c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.900241 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3d7aed46234a9095f0013f8ab219c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.900741 2352 raft_consensus.cc:2267] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.901208 749 tablet_service.cc:1467] Processing CreateTablet for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e (DEFAULT_TABLE table=default.Table_9ea3073c95ce42d6b347ad6af5eb8f0a [id=11bd8bb10a1d421bbddd9885b267c522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.902765 2352 ts_tablet_manager.cc:1905] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.904227 881 tablet_service.cc:1467] Processing CreateTablet for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e (DEFAULT_TABLE table=default.Table_9ea3073c95ce42d6b347ad6af5eb8f0a [id=11bd8bb10a1d421bbddd9885b267c522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.904528 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.905934 886 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8527249404ac4be01021778e64fbe (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.906236 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8527249404ac4be01021778e64fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.907500 890 tablet_service.cc:1467] Processing CreateTablet for tablet 587e4962bcc44e7ebcce808beb2a8acf (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.907750 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587e4962bcc44e7ebcce808beb2a8acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.908565 750 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8527249404ac4be01021778e64fbe (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.909189 747 tablet_service.cc:1467] Processing CreateTablet for tablet 587e4962bcc44e7ebcce808beb2a8acf (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:30.909647 853 consensus_peers.cc:487] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d424b4da66184bd19eccefad80621516. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.910271 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c151d25b1714b2d86e83e73a58b7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.910668 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 147) took 6625 ms. Trace:
I20250110 02:51:30.910737 2357 rpcz_store.cc:276] 0110 02:51:24.285086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.285135 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:30.617188 (+6332053us) ts_tablet_manager.cc:1145] Deleting tablet 7564208da99449cd97338a41b263e750
0110 02:51:30.617197 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.769035 (+151838us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.799422 (+ 30387us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.910651 (+111229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6331911}
I20250110 02:51:30.911098 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 152) took 5986 ms. Trace:
I20250110 02:51:30.911370 2357 rpcz_store.cc:276] 0110 02:51:24.924280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.924327 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:30.910984 (+5986657us) ts_tablet_manager.cc:1145] Deleting tablet b45b91d084a44c1e8cdf1553967878e2
0110 02:51:30.910992 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.911069 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5975943}
I20250110 02:51:30.911615 2357 tablet_replica.cc:331] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.911779 2357 raft_consensus.cc:2238] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.911909 2357 raft_consensus.cc:2267] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.912408 2357 ts_tablet_manager.cc:1905] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.914187 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1885ab76b5c424d82fddd6b5481e4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.914527 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7564208da99449cd97338a41b263e750 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]) successfully deleted
W20250110 02:51:30.914820 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b45b91d084a44c1e8cdf1553967878e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b45b91d084a44c1e8cdf1553967878e2 already in progress: creating tablet
I20250110 02:51:30.920125 895 tablet_service.cc:1467] Processing CreateTablet for tablet 86d96c45f70a468a91ea6f4694e7877f (DEFAULT_TABLE table=default.Table_627f1a96679c4cad8d88b188e4d769b8 [id=744db20489294f98ad2a37119678c2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.920284 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8435d5dd16b146c98ae86293a074445f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.920542 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d96c45f70a468a91ea6f4694e7877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.922542 2357 ts_tablet_manager.cc:1918] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.923061 1184 tablet_bootstrap.cc:492] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.923148 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8ff3f5bb2f4240837130df790fa6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.926596 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e28d8769f364e10aad54b60fb221f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.926993 1184 tablet_bootstrap.cc:654] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.922787 2357 log.cc:1198] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1edb4e06e40c43438882e1eb622d788e
I20250110 02:51:30.936373 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e25b2b512e6405b956d49944c418c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.939409 1188 tablet_bootstrap.cc:654] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.938295 2357 ts_tablet_manager.cc:1939] T 1edb4e06e40c43438882e1eb622d788e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:30.867936 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 8435d5dd16b146c98ae86293a074445f (DEFAULT_TABLE table=default.Table_9883b2a0f5e448df83b62a280a1cb915 [id=1fb43093bd654d8e94fb42cdfa0e565b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.868481 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 7c151d25b1714b2d86e83e73a58b7b49 (DEFAULT_TABLE table=default.Table_3fbaaf982d4c443a8d174437ef013ab2 [id=602370f8e86e4145a05c8b9978c25ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.868858 1038 tablet_service.cc:1467] Processing CreateTablet for tablet c1885ab76b5c424d82fddd6b5481e4d6 (DEFAULT_TABLE table=default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.869282 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d96c45f70a468a91ea6f4694e7877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.942389 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd669ba7a4bf47b39063f61d33556724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.883045 1039 tablet_service.cc:1467] Processing CreateTablet for tablet af8ff3f5bb2f4240837130df790fa6d8 (DEFAULT_TABLE table=default.Table_2433934279a049aca30f6cccd6c71952 [id=2ee5bd916ba84d42aa04b8991e381465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.942883 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8ff3f5bb2f4240837130df790fa6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.887835 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 9e28d8769f364e10aad54b60fb221f35 (DEFAULT_TABLE table=default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.946521 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e28d8769f364e10aad54b60fb221f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.946678 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3d7aed46234a9095f0013f8ab219c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.890010 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0e25b2b512e6405b956d49944c418c19 (DEFAULT_TABLE table=default.Table_06192f79203e4dd4bda8cda74e08e7c7 [id=8af0a73455e54ea5b8ba1872c3238b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.950645 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.950739 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e25b2b512e6405b956d49944c418c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.854717 1181 consensus_queue.cc:260] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.893034 1031 tablet_service.cc:1467] Processing CreateTablet for tablet bd669ba7a4bf47b39063f61d33556724 (DEFAULT_TABLE table=default.Table_2d3082a17dfd4f50b025f2da061be352 [id=4e490b91659d4526b1dbb0841112b55f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.954034 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd669ba7a4bf47b39063f61d33556724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.956383 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8527249404ac4be01021778e64fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.899673 1037 tablet_service.cc:1467] Processing CreateTablet for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e (DEFAULT_TABLE table=default.Table_9ea3073c95ce42d6b347ad6af5eb8f0a [id=11bd8bb10a1d421bbddd9885b267c522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.908924 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 587e4962bcc44e7ebcce808beb2a8acf (DEFAULT_TABLE table=default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.958956 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b39772cd4c74e2cbe7b11fd8c2eb54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.959162 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587e4962bcc44e7ebcce808beb2a8acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.961843 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b9b579176164c7a8aeeab632d764670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.962247 824 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:30.962687 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c151d25b1714b2d86e83e73a58b7b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.966599 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8435d5dd16b146c98ae86293a074445f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.970456 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1885ab76b5c424d82fddd6b5481e4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.971086 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d96c45f70a468a91ea6f4694e7877f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:30.940327 994 consensus_peers.cc:487] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:30.857347 996 leader_election.cc:304] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
W20250110 02:51:30.974188 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 116864us
I20250110 02:51:30.974738 2898 raft_consensus.cc:2798] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:30.974848 2898 raft_consensus.cc:695] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:30.975059 2898 consensus_queue.cc:237] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.872591 2224 ts_tablet_manager.cc:1918] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.975925 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 153) took 6012 ms. Trace:
I20250110 02:51:30.975979 2224 log.cc:1198] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a81c031451104d5a93a1672b4b1918c5
I20250110 02:51:30.976009 2357 rpcz_store.cc:276] 0110 02:51:24.963797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.963853 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:30.911575 (+5947722us) ts_tablet_manager.cc:1145] Deleting tablet 1edb4e06e40c43438882e1eb622d788e
0110 02:51:30.911582 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.915838 (+ 4256us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.922409 (+ 6571us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.975906 (+ 53497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5947527}
I20250110 02:51:30.976310 2224 ts_tablet_manager.cc:1939] T a81c031451104d5a93a1672b4b1918c5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:30.976361 2357 tablet_replica.cc:331] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:30.976572 2357 raft_consensus.cc:2238] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.976555 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1edb4e06e40c43438882e1eb622d788e (table default.Table_658b8237dabd45f1a02c1fb92070f6e0 [id=387a7fedac2a4e4998db6500fc8c9334]) successfully deleted
I20250110 02:51:30.976598 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 220) took 5959 ms. Trace:
I20250110 02:51:30.976706 2357 raft_consensus.cc:2267] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.976666 2224 rpcz_store.cc:276] 0110 02:51:25.017088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.017126 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:30.777221 (+5760095us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:30.777231 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.827840 (+ 50609us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.872495 (+ 44655us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.976583 (+104088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5759964}
I20250110 02:51:30.976984 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 221) took 5959 ms. Trace:
I20250110 02:51:30.977048 2224 rpcz_store.cc:276] 0110 02:51:25.017189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.017276 (+ 87us) service_pool.cc:225] Handling call
0110 02:51:30.976884 (+5959608us) ts_tablet_manager.cc:1145] Deleting tablet 8e79175853474cbea951f5a5686bf630
0110 02:51:30.976911 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.976970 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5959504}
I20250110 02:51:30.977216 2357 ts_tablet_manager.cc:1905] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.977273 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 217) took 5960 ms. Trace:
I20250110 02:51:30.977324 2224 rpcz_store.cc:276] 0110 02:51:25.016748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.017802 (+ 1054us) service_pool.cc:225] Handling call
0110 02:51:30.977215 (+5959413us) ts_tablet_manager.cc:1145] Deleting tablet 240b64d551a548e884ee2a7238c0363a
0110 02:51:30.977222 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.977263 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5959284}
I20250110 02:51:30.977547 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 224) took 5960 ms. Trace:
I20250110 02:51:30.977595 2224 rpcz_store.cc:276] 0110 02:51:25.017454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.017837 (+ 383us) service_pool.cc:225] Handling call
0110 02:51:30.977496 (+5959659us) ts_tablet_manager.cc:1145] Deleting tablet 35ec1dba54a247d5ba9f9e053aa3b645
0110 02:51:30.977502 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.977537 (+ 35us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32,"tablet-delete.queue_time_us":5959565}
I20250110 02:51:30.977797 2224 tablet_replica.cc:331] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:30.977962 2224 raft_consensus.cc:2238] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.978067 2224 raft_consensus.cc:2267] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.897781 1035 tablet_service.cc:1467] Processing CreateTablet for tablet 1c3d7aed46234a9095f0013f8ab219c1 (DEFAULT_TABLE table=default.Table_acfdadc9edbf4eaf94f0a667273d93ec [id=1aae6180ec594da591e444f6cd4f04c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.980028 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3d7aed46234a9095f0013f8ab219c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.904690 1041 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8527249404ac4be01021778e64fbe (DEFAULT_TABLE table=default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:30.981524 1184 tablet_bootstrap.cc:492] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:30.981626 1184 ts_tablet_manager.cc:1397] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.002s
I20250110 02:51:30.983018 2352 ts_tablet_manager.cc:1918] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:30.983129 2352 log.cc:1198] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/19e4af23720048fe98b3231ba402d45c
I20250110 02:51:30.983474 2352 ts_tablet_manager.cc:1939] T 19e4af23720048fe98b3231ba402d45c P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:30.984898 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 212) took 4669 ms. Trace:
I20250110 02:51:30.985412 2352 rpcz_store.cc:276] 0110 02:51:26.315074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315167 (+ 93us) service_pool.cc:225] Handling call
0110 02:51:30.897043 (+4581876us) ts_tablet_manager.cc:1145] Deleting tablet 19e4af23720048fe98b3231ba402d45c
0110 02:51:30.897051 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.920210 (+ 23159us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.982918 (+ 62708us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.984880 (+ 1962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4581741}
I20250110 02:51:30.986076 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 19e4af23720048fe98b3231ba402d45c (table default.Table_9ca42c50470d4cebaf832606cb958d23 [id=5e514acaba2d498a8724032d6bbc65d9]) successfully deleted
I20250110 02:51:30.986322 2352 tablet_replica.cc:331] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:30.986506 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d8527249404ac4be01021778e64fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:30.986654 1184 raft_consensus.cc:357] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.986755 2352 raft_consensus.cc:2238] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:30.986835 1184 raft_consensus.cc:383] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.986899 1184 raft_consensus.cc:738] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.986924 2352 raft_consensus.cc:2267] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:30.987069 1184 consensus_queue.cc:260] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:30.987502 1184 ts_tablet_manager.cc:1428] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250110 02:51:30.987716 1184 tablet_bootstrap.cc:492] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:30.988186 2352 ts_tablet_manager.cc:1905] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.988958 1184 tablet_bootstrap.cc:654] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:30.989903 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a81c031451104d5a93a1672b4b1918c5 (table default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]) successfully deleted
W20250110 02:51:30.990159 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because the tablet was not found. No further retry: Not found: Tablet not found: 8e79175853474cbea951f5a5686bf630
W20250110 02:51:30.990345 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because the tablet was not found. No further retry: Not found: Tablet not found: 240b64d551a548e884ee2a7238c0363a
W20250110 02:51:30.990588 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 35ec1dba54a247d5ba9f9e053aa3b645 because the tablet was not found. No further retry: Not found: Tablet not found: 35ec1dba54a247d5ba9f9e053aa3b645
I20250110 02:51:30.991046 1181 ts_tablet_manager.cc:1428] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.148s user 0.002s sys 0.000s
I20250110 02:51:30.992048 1181 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:30.993238 2478 consensus_queue.cc:1035] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:30.995304 2224 ts_tablet_manager.cc:1905] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:30.995666 1188 tablet_bootstrap.cc:492] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:30.995787 1188 ts_tablet_manager.cc:1397] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20250110 02:51:30.996414 1188 raft_consensus.cc:357] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:30.996887 1188 raft_consensus.cc:383] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:30.998064 1188 raft_consensus.cc:738] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:30.996549 2352 ts_tablet_manager.cc:1918] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.002125 2352 log.cc:1198] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c5601ca24b8f428e8a51cbeec3664901
I20250110 02:51:31.002557 2352 ts_tablet_manager.cc:1939] T c5601ca24b8f428e8a51cbeec3664901 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.001993 1188 consensus_queue.cc:260] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.003190 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 213) took 4688 ms. Trace:
I20250110 02:51:31.003618 2352 rpcz_store.cc:276] 0110 02:51:26.315158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315318 (+ 160us) service_pool.cc:225] Handling call
0110 02:51:30.986134 (+4670816us) ts_tablet_manager.cc:1145] Deleting tablet c5601ca24b8f428e8a51cbeec3664901
0110 02:51:30.986144 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.993799 (+ 7655us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:30.996479 (+ 2680us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.003175 (+ 6696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4670709}
I20250110 02:51:31.004232 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c5601ca24b8f428e8a51cbeec3664901 (table default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]) successfully deleted
I20250110 02:51:31.004308 2352 tablet_replica.cc:331] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.004906 2352 raft_consensus.cc:2238] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.005127 2352 raft_consensus.cc:2267] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.005879 2352 ts_tablet_manager.cc:1905] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.006662 1184 tablet_bootstrap.cc:492] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.006805 1184 ts_tablet_manager.cc:1397] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250110 02:51:31.007505 1181 tablet_bootstrap.cc:654] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.007704 1184 raft_consensus.cc:357] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.007911 1184 raft_consensus.cc:383] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.008003 1184 raft_consensus.cc:738] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.008287 1184 consensus_queue.cc:260] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.008165 1188 ts_tablet_manager.cc:1428] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20250110 02:51:31.008913 1184 ts_tablet_manager.cc:1428] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:31.009299 1184 tablet_bootstrap.cc:492] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.010670 1188 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.015738 2352 ts_tablet_manager.cc:1918] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.018496 2352 log.cc:1198] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:31.018987 2352 ts_tablet_manager.cc:1939] T 4d970e2fe72b4727968ca605b3e772cc P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.020200 2224 ts_tablet_manager.cc:1918] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.020243 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 215) took 4704 ms. Trace:
I20250110 02:51:31.020344 2224 log.cc:1198] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7e554d00d05b4fa7a14c9b6159f9d726
I20250110 02:51:31.020354 2352 rpcz_store.cc:276] 0110 02:51:26.315331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315447 (+ 116us) service_pool.cc:225] Handling call
0110 02:51:31.004265 (+4688818us) ts_tablet_manager.cc:1145] Deleting tablet 4d970e2fe72b4727968ca605b3e772cc
0110 02:51:31.004275 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.009812 (+ 5537us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.015640 (+ 5828us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.020228 (+ 4588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":204,"tablet-delete.queue_time_us":4688682}
I20250110 02:51:31.020628 2224 ts_tablet_manager.cc:1939] T 7e554d00d05b4fa7a14c9b6159f9d726 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.020870 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 4d970e2fe72b4727968ca605b3e772cc (table default.Table_5bf91a5e0e004e1880fdf025d5d9ba38 [id=c584ebe680bb449e92795453c9b6ab74]) successfully deleted
I20250110 02:51:31.020985 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 217) took 4705 ms. Trace:
I20250110 02:51:31.021050 2352 rpcz_store.cc:276] 0110 02:51:26.315481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315608 (+ 127us) service_pool.cc:225] Handling call
0110 02:51:31.020901 (+4705293us) ts_tablet_manager.cc:1145] Deleting tablet 53ae5a6572bd4e7e90e92d4b1cf8d820
0110 02:51:31.020909 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.020973 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4705189}
I20250110 02:51:31.021314 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 226) took 6003 ms. Trace:
I20250110 02:51:31.021438 2224 rpcz_store.cc:276] 0110 02:51:25.017633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.017942 (+ 309us) service_pool.cc:225] Handling call
0110 02:51:30.977766 (+5959824us) ts_tablet_manager.cc:1145] Deleting tablet 7e554d00d05b4fa7a14c9b6159f9d726
0110 02:51:30.977772 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:30.998173 (+ 20401us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.020111 (+ 21938us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.021292 (+ 1181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5953160}
W20250110 02:51:31.021785 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because the tablet was not found. No further retry: Not found: Tablet not found: 53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:31.021836 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 219) took 4706 ms. Trace:
I20250110 02:51:31.021916 2352 rpcz_store.cc:276] 0110 02:51:26.315653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315731 (+ 78us) service_pool.cc:225] Handling call
0110 02:51:31.021748 (+4706017us) ts_tablet_manager.cc:1145] Deleting tablet 8d1f2acd39be40e4b71cc0dcf68e7567
0110 02:51:31.021755 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.021824 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4705912}
I20250110 02:51:31.022207 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7e554d00d05b4fa7a14c9b6159f9d726 (table default.Table_f45dec555ce747f9be01f55fa8d8383f [id=953ce2773b0041909b2e1f7767f361bb]) successfully deleted
W20250110 02:51:31.022624 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
I20250110 02:51:31.022651 2352 tablet_replica.cc:331] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.022818 2352 raft_consensus.cc:2238] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.022850 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 214) took 6007 ms. Trace:
I20250110 02:51:31.022923 2224 rpcz_store.cc:276] 0110 02:51:25.015441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.015640 (+ 199us) service_pool.cc:225] Handling call
0110 02:51:31.022478 (+6006838us) ts_tablet_manager.cc:1145] Deleting tablet 76833ce515cf4ee9bc8fff0f626af220
0110 02:51:31.022486 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.022830 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":653,"tablet-delete.queue_time_us":5998496}
W20250110 02:51:31.023356 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because the tablet was not found. No further retry: Not found: Tablet not found: 76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:31.023381 2352 raft_consensus.cc:2267] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.023669 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 232) took 5948 ms. Trace:
I20250110 02:51:31.023789 2224 rpcz_store.cc:276] 0110 02:51:25.075480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.075528 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:31.023566 (+5948038us) ts_tablet_manager.cc:1145] Deleting tablet 4aa78a9ef5eb443da0c8983a7190440b
0110 02:51:31.023576 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.023650 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5947536}
W20250110 02:51:31.024168 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:31.024349 2352 ts_tablet_manager.cc:1905] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.024443 2224 tablet_replica.cc:331] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:31.025920 1184 tablet_bootstrap.cc:654] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.032099 2352 ts_tablet_manager.cc:1918] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.035648 2352 log.cc:1198] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d761ef0d1e5542b1979c29bb2d579489
I20250110 02:51:31.036131 2352 ts_tablet_manager.cc:1939] T d761ef0d1e5542b1979c29bb2d579489 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.036715 2345 raft_consensus.cc:491] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.036720 2952 raft_consensus.cc:491] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.036839 2952 raft_consensus.cc:513] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.036868 2345 raft_consensus.cc:513] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.037468 2345 leader_election.cc:290] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.037482 2952 leader_election.cc:290] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.038324 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 220) took 4722 ms. Trace:
I20250110 02:51:31.038497 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.039077 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e35fe84c1e04d3eb295a6d8d5d40ea4" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.040663 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.040840 994 consensus_peers.cc:487] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.040972 2504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: configured and running, proceeding with master startup.
W20250110 02:51:31.041452 711 leader_election.cc:343] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.039808 853 leader_election.cc:304] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:31.041909 2345 raft_consensus.cc:2743] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.041185 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.041775 2352 rpcz_store.cc:276] 0110 02:51:26.315738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315860 (+ 122us) service_pool.cc:225] Handling call
0110 02:51:31.022616 (+4706756us) ts_tablet_manager.cc:1145] Deleting tablet d761ef0d1e5542b1979c29bb2d579489
0110 02:51:31.022623 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.029509 (+ 6886us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.032017 (+ 2508us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.038307 (+ 6290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4706640}
I20250110 02:51:31.042281 2352 tablet_replica.cc:331] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.042515 2352 raft_consensus.cc:2238] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:31.042560 709 leader_election.cc:343] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.042640 709 leader_election.cc:304] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.042788 2352 pending_rounds.cc:62] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Trying to abort 1 pending ops.
I20250110 02:51:31.039402 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:31.042830 2352 pending_rounds.cc:69] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611795671920640 op_type: NO_OP noop_request { }
I20250110 02:51:31.042878 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:31.042943 2352 raft_consensus.cc:2883] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:31.043030 2352 raft_consensus.cc:2267] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.043040 2952 raft_consensus.cc:2743] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.043565 2352 ts_tablet_manager.cc:1905] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.044107 1184 tablet_bootstrap.cc:492] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.044349 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587e4962bcc44e7ebcce808beb2a8acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.044261 1184 ts_tablet_manager.cc:1397] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20250110 02:51:31.045015 1184 raft_consensus.cc:357] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.045217 1184 raft_consensus.cc:383] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.045300 1184 raft_consensus.cc:738] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.045552 1184 consensus_queue.cc:260] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.045768 2424 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 3 FOLLOWER]: Advancing to term 4
I20250110 02:51:31.046164 1184 ts_tablet_manager.cc:1428] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:31.048781 2352 ts_tablet_manager.cc:1918] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.053404 2352 log.cc:1198] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/65285bce60fd4015b085deb7442eea6e
I20250110 02:51:31.051149 1188 tablet_bootstrap.cc:654] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.053941 2352 ts_tablet_manager.cc:1939] T 65285bce60fd4015b085deb7442eea6e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.055231 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 222) took 4739 ms. Trace:
I20250110 02:51:31.055652 1181 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.055706 2224 raft_consensus.cc:2238] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.055855 1181 ts_tablet_manager.cc:1397] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.000s
I20250110 02:51:31.056485 2352 rpcz_store.cc:276] 0110 02:51:26.315886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.315997 (+ 111us) service_pool.cc:225] Handling call
0110 02:51:31.042239 (+4726242us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:31.042250 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.046226 (+ 3976us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.048699 (+ 2473us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.055210 (+ 6511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4726130}
I20250110 02:51:31.056860 1181 raft_consensus.cc:357] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.057307 2352 tablet_replica.cc:331] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.057330 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 65285bce60fd4015b085deb7442eea6e (table default.Table_ad6ed08b6a0e47599bd592fbaf2c5090 [id=d5b455f5292045699c4ff5c254d68acd]) successfully deleted
I20250110 02:51:31.057701 2352 raft_consensus.cc:2238] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.058058 2352 raft_consensus.cc:2267] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.058531 1181 raft_consensus.cc:383] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.058655 1181 raft_consensus.cc:738] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.058794 2352 ts_tablet_manager.cc:1905] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.060442 1184 tablet_bootstrap.cc:492] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:31.059795 994 consensus_peers.cc:487] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5601ca24b8f428e8a51cbeec3664901. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.061985 1181 consensus_queue.cc:260] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.061620 2952 raft_consensus.cc:491] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.062489 2952 raft_consensus.cc:513] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.063166 2952 leader_election.cc:290] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.063630 1184 tablet_bootstrap.cc:654] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.063591 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.063701 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.064040 1181 ts_tablet_manager.cc:1428] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250110 02:51:31.064064 711 leader_election.cc:343] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:31.064225 709 leader_election.cc:343] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.064340 1181 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.064328 709 leader_election.cc:304] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.064708 2952 raft_consensus.cc:2743] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.066799 1181 tablet_bootstrap.cc:654] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.070024 2352 ts_tablet_manager.cc:1918] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.073002 2352 log.cc:1198] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/240b64d551a548e884ee2a7238c0363a
I20250110 02:51:31.073467 2352 ts_tablet_manager.cc:1939] T 240b64d551a548e884ee2a7238c0363a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.075500 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 224) took 4759 ms. Trace:
I20250110 02:51:31.076105 2352 rpcz_store.cc:276] 0110 02:51:26.316053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316124 (+ 71us) service_pool.cc:225] Handling call
0110 02:51:31.057261 (+4741137us) ts_tablet_manager.cc:1145] Deleting tablet 240b64d551a548e884ee2a7238c0363a
0110 02:51:31.057272 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.062937 (+ 5665us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.069940 (+ 7003us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.075481 (+ 5541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4741030}
I20250110 02:51:31.076793 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 240b64d551a548e884ee2a7238c0363a (table default.Table_23df132302f5414db477144cbb57b923 [id=1547d15656654893888a22f7ce34de0b]) successfully deleted
I20250110 02:51:31.077186 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 225) took 4760 ms. Trace:
I20250110 02:51:31.077536 2352 rpcz_store.cc:276] 0110 02:51:26.316148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316251 (+ 103us) service_pool.cc:225] Handling call
0110 02:51:31.076815 (+4760564us) ts_tablet_manager.cc:1145] Deleting tablet 28340f0f9a2043a199c74f13755eb702
0110 02:51:31.076824 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.077109 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4760417}
W20250110 02:51:31.078142 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because the tablet was not found. No further retry: Not found: Tablet not found: 28340f0f9a2043a199c74f13755eb702
I20250110 02:51:31.078441 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 227) took 4762 ms. Trace:
I20250110 02:51:31.078640 2352 rpcz_store.cc:276] 0110 02:51:26.316327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316419 (+ 92us) service_pool.cc:225] Handling call
0110 02:51:31.078156 (+4761737us) ts_tablet_manager.cc:1145] Deleting tablet f9b379b357194d729108428cbaebd51a
0110 02:51:31.078165 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.078426 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4761635}
I20250110 02:51:31.079351 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 229) took 4762 ms. Trace:
W20250110 02:51:31.079377 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f9b379b357194d729108428cbaebd51a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b379b357194d729108428cbaebd51a already in progress: creating tablet
I20250110 02:51:31.079668 2352 rpcz_store.cc:276] 0110 02:51:26.316482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316542 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:31.079277 (+4762735us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:31.079286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.079338 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4762599}
W20250110 02:51:31.080525 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because the tablet was not found. No further retry: Not found: Tablet not found: b375a932156d4abb908dbf327689b0f9
I20250110 02:51:31.078218 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:31.080929 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:31.081447 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 230) took 4764 ms. Trace:
I20250110 02:51:31.081539 2352 rpcz_store.cc:276] 0110 02:51:26.316570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316703 (+ 133us) service_pool.cc:225] Handling call
0110 02:51:31.080163 (+4763460us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:31.081384 (+ 1221us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.081434 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4763342}
I20250110 02:51:31.060307 2224 raft_consensus.cc:2267] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.081919 2352 tablet_replica.cc:331] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.081648 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:31.082152 2352 raft_consensus.cc:2238] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.082307 2352 raft_consensus.cc:2267] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.082859 2352 ts_tablet_manager.cc:1905] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.082942 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:31.083648 2357 ts_tablet_manager.cc:1918] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 02:51:31.084045 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because the tablet was not found. No further retry: Not found: Tablet not found: d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:31.084507 1450 catalog_manager.cc:5526] T 6479d71a5142436b8c6ea2f76c221b33 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.086771 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:31.086941 2357 log.cc:1198] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/818dcbf83303412d97b097913bcd36c3
I20250110 02:51:31.087263 2357 ts_tablet_manager.cc:1939] T 818dcbf83303412d97b097913bcd36c3 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.087503 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:31.088284 1188 tablet_bootstrap.cc:492] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.088457 1188 ts_tablet_manager.cc:1397] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20250110 02:51:31.088492 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:31.088781 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:31.089215 1188 raft_consensus.cc:357] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.089419 1188 raft_consensus.cc:383] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.089488 1188 raft_consensus.cc:738] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.087553 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 155) took 6099 ms. Trace:
I20250110 02:51:31.089701 1188 consensus_queue.cc:260] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.090276 1188 ts_tablet_manager.cc:1428] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:31.090307 2906 consensus_queue.cc:1035] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.093760 2951 raft_consensus.cc:491] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.093880 2951 raft_consensus.cc:513] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.094492 2951 leader_election.cc:290] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.095054 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28f191afa2a43608bbd74322bec97a6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.095592 711 leader_election.cc:343] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.093799 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:31.095676 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28f191afa2a43608bbd74322bec97a6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.095939 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:31.096819 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
W20250110 02:51:31.097252 709 leader_election.cc:343] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.097364 709 leader_election.cc:304] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.097528 2952 raft_consensus.cc:2743] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.098870 2224 ts_tablet_manager.cc:1905] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.100431 2456 consensus_queue.cc:1035] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.101868 2352 ts_tablet_manager.cc:1918] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.103377 2352 log.cc:1198] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:31.097940 1450 catalog_manager.cc:5526] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.103993 2352 ts_tablet_manager.cc:1939] T c2b2aa3a302a47d3a3cb4dfc2e330cfa P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.104403 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:31.106871 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 207) took 4792 ms. Trace:
I20250110 02:51:31.106994 2352 rpcz_store.cc:276] 0110 02:51:26.314705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316750 (+ 2045us) service_pool.cc:225] Handling call
0110 02:51:31.081868 (+4765118us) ts_tablet_manager.cc:1145] Deleting tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
0110 02:51:31.081877 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.085786 (+ 3909us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.101787 (+ 16001us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.106856 (+ 5069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4764975}
I20250110 02:51:31.107199 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:31.107436 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa (table default.Table_3ceaee2d997344448ebb32110efcfcf2 [id=be5eb400d67c4aa881f39dd7c9b44b88]) successfully deleted
I20250110 02:51:31.107564 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 231) took 4790 ms. Trace:
I20250110 02:51:31.107656 2352 rpcz_store.cc:276] 0110 02:51:26.316691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316840 (+ 149us) service_pool.cc:225] Handling call
0110 02:51:31.107451 (+4790611us) ts_tablet_manager.cc:1145] Deleting tablet 5af3d61366b2432db53b18f2b7bb49d9
0110 02:51:31.107460 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.107550 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4790490}
W20250110 02:51:31.108033 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5af3d61366b2432db53b18f2b7bb49d9 because the tablet was not found. No further retry: Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:31.108070 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 209) took 4793 ms. Trace:
I20250110 02:51:31.107729 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:31.108204 2352 rpcz_store.cc:276] 0110 02:51:26.314859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.316923 (+ 2064us) service_pool.cc:225] Handling call
0110 02:51:31.108005 (+4791082us) ts_tablet_manager.cc:1145] Deleting tablet d6fd370138ac4039be55642f1b4c5607
0110 02:51:31.108013 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.108058 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4790954}
W20250110 02:51:31.108582 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because the tablet was not found. No further retry: Not found: Tablet not found: d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:31.109234 2352 tablet_replica.cc:331] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.109640 2352 raft_consensus.cc:2238] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.109887 1184 tablet_bootstrap.cc:492] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.110363 1188 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.109694 1450 catalog_manager.cc:5526] T b12ce7a0dc4f4f3586a2f918d31234e1 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.110553 1184 ts_tablet_manager.cc:1397] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250110 02:51:31.110944 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:31.111258 1184 raft_consensus.cc:357] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.111573 1184 raft_consensus.cc:383] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.111639 1184 raft_consensus.cc:738] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.111832 1184 consensus_queue.cc:260] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.112121 1188 tablet_bootstrap.cc:654] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.112294 1184 ts_tablet_manager.cc:1428] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:31.113091 2352 raft_consensus.cc:2267] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.113785 2352 ts_tablet_manager.cc:1905] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.117131 2224 ts_tablet_manager.cc:1918] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.089593 2357 rpcz_store.cc:276] 0110 02:51:24.987588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.987642 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:30.976316 (+5988674us) ts_tablet_manager.cc:1145] Deleting tablet 818dcbf83303412d97b097913bcd36c3
0110 02:51:30.976326 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.050132 (+ 73806us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.083557 (+ 33425us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.087539 (+ 3982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5988460}
I20250110 02:51:31.117602 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 818dcbf83303412d97b097913bcd36c3 (table default.Table_c2fd603cae0f4558877bff141025cd02 [id=cf91ae38d64542a5b7762867fade7a91]) successfully deleted
I20250110 02:51:31.118011 1450 catalog_manager.cc:5526] T 9d3e52914a424cb3a8bf422b3a56fb22 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.118536 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 154) took 6131 ms. Trace:
I20250110 02:51:31.118647 2357 rpcz_store.cc:276] 0110 02:51:24.987429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:24.987892 (+ 463us) service_pool.cc:225] Handling call
0110 02:51:31.117902 (+6130010us) ts_tablet_manager.cc:1145] Deleting tablet 24034fb3da6e47978cdc9b02e714e30d
0110 02:51:31.118333 (+ 431us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.118514 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6129867}
W20250110 02:51:31.119118 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 24034fb3da6e47978cdc9b02e714e30d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24034fb3da6e47978cdc9b02e714e30d already in progress: creating tablet
I20250110 02:51:31.119243 1184 tablet_bootstrap.cc:492] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.119405 2357 tablet_replica.cc:331] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:31.119598 2357 raft_consensus.cc:2238] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.119746 2357 raft_consensus.cc:2267] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.120021 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:31.120337 2357 ts_tablet_manager.cc:1905] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.121976 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09fd9734714a4847ab2a4ebc37af8960 (table default.Table_6aaaf7ba453a4e74a3a55367388c0584 [id=ee9fe75dbd374159bb7a40bdc3576d06]). Expected version 3 got 0
I20250110 02:51:31.123111 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:31.127104 1184 tablet_bootstrap.cc:654] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.127580 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:31.128223 2352 ts_tablet_manager.cc:1918] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.129485 2352 log.cc:1198] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:31.130193 2352 ts_tablet_manager.cc:1939] T 5de9cf980d9c4117b4ec3ca937cfcf01 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.131872 1188 tablet_bootstrap.cc:492] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.132079 1188 ts_tablet_manager.cc:1397] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250110 02:51:31.132200 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:31.132581 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:31.132915 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:31.133224 1188 raft_consensus.cc:357] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.133507 1188 raft_consensus.cc:383] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.133580 1188 raft_consensus.cc:738] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.133790 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:31.133843 1188 consensus_queue.cc:260] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.142079 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:31.142740 1188 ts_tablet_manager.cc:1428] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:31.142992 2951 raft_consensus.cc:491] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.143013 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 211) took 4828 ms. Trace:
I20250110 02:51:31.143078 2224 log.cc:1198] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7564208da99449cd97338a41b263e750
I20250110 02:51:31.143092 2352 rpcz_store.cc:276] 0110 02:51:26.315002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317078 (+ 2076us) service_pool.cc:225] Handling call
0110 02:51:31.108568 (+4791490us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:31.108576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.122864 (+ 14288us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.128152 (+ 5288us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.142996 (+ 14844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4791378}
I20250110 02:51:31.143110 2951 raft_consensus.cc:513] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.143360 2352 tablet_replica.cc:331] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.143457 2224 ts_tablet_manager.cc:1939] T 7564208da99449cd97338a41b263e750 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.143494 2352 raft_consensus.cc:2238] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.143684 2951 leader_election.cc:290] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.143785 2352 raft_consensus.cc:2267] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.144040 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 5de9cf980d9c4117b4ec3ca937cfcf01 (table default.Table_7b5def1fa803454baf15dfee9b48fe75 [id=29a95fe25bd54dd59a9c658024fd21ac]) successfully deleted
I20250110 02:51:31.144207 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 233) took 6068 ms. Trace:
I20250110 02:51:31.144325 2352 ts_tablet_manager.cc:1905] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.144280 2224 rpcz_store.cc:276] 0110 02:51:25.076201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.076262 (+ 61us) service_pool.cc:225] Handling call
0110 02:51:31.024407 (+5948145us) ts_tablet_manager.cc:1145] Deleting tablet 7564208da99449cd97338a41b263e750
0110 02:51:31.024414 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.056064 (+ 31650us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x616000146854. stack: 00007f56d1ae3fd6 00007f56d1ae3d90 00007f56d0ac348d 000000000096d544 000000000096d50d 000000000096ceda 00007f56dd386ed0 00007f56de67b463 00007f56e111feb7 00007f56e1152584 00007f56e1133690 00007f56e113341d 0000000000a71ecc 00007f56d1b2ab64 00007f56d1b2e18b 00007f56d1b2e008
0110 02:51:31.114258 (+ 58194us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.117060 (+ 2802us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.144187 (+ 27127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71390080,"tablet-delete.queue_time_us":5947736}
I20250110 02:51:31.144493 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.144629 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 235) took 6067 ms. Trace:
I20250110 02:51:31.144686 2224 rpcz_store.cc:276] 0110 02:51:25.077530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.079613 (+ 2083us) service_pool.cc:225] Handling call
0110 02:51:31.144520 (+6064907us) ts_tablet_manager.cc:1145] Deleting tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
0110 02:51:31.144528 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.144616 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6064775}
I20250110 02:51:31.144974 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 236) took 5747 ms. Trace:
I20250110 02:51:31.145035 2224 rpcz_store.cc:276] 0110 02:51:25.397923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.398314 (+ 391us) service_pool.cc:225] Handling call
0110 02:51:31.144885 (+5746571us) ts_tablet_manager.cc:1145] Deleting tablet fe860d60b3194dc5a2e7a94d39d4b8dd
0110 02:51:31.144893 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.144959 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5746356}
I20250110 02:51:31.145068 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.145267 2224 tablet_replica.cc:331] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:31.145447 2224 raft_consensus.cc:2238] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.145726 2224 raft_consensus.cc:2267] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.145732 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7564208da99449cd97338a41b263e750 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]) successfully deleted
W20250110 02:51:31.145964 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a already in progress: creating tablet
W20250110 02:51:31.146123 994 consensus_peers.cc:487] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:31.146194 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
I20250110 02:51:31.146222 2224 ts_tablet_manager.cc:1905] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:31.147042 711 leader_election.cc:343] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:31.147686 709 leader_election.cc:343] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.147775 709 leader_election.cc:304] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.148547 2951 raft_consensus.cc:2743] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.148525 2898 consensus_queue.cc:1035] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.150300 2352 ts_tablet_manager.cc:1918] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.150455 2352 log.cc:1198] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:31.150830 2352 ts_tablet_manager.cc:1939] T 9289de0630804f9aa60fdaf5f3f15f68 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.151567 2224 ts_tablet_manager.cc:1918] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.151683 2224 log.cc:1198] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/6555b0a4491046bf8f5cd4ee406c3955
I20250110 02:51:31.151981 2224 ts_tablet_manager.cc:1939] T 6555b0a4491046bf8f5cd4ee406c3955 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.153214 1181 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.152029 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:31.153352 1181 ts_tablet_manager.cc:1397] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20250110 02:51:31.151607 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 214) took 4836 ms. Trace:
I20250110 02:51:31.153997 1181 raft_consensus.cc:357] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.154176 1181 raft_consensus.cc:383] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.154240 1181 raft_consensus.cc:738] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.153898 2352 rpcz_store.cc:276] 0110 02:51:26.315238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317214 (+ 1976us) service_pool.cc:225] Handling call
0110 02:51:31.143324 (+4826110us) ts_tablet_manager.cc:1145] Deleting tablet 9289de0630804f9aa60fdaf5f3f15f68
0110 02:51:31.143332 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.147583 (+ 4251us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.150231 (+ 2648us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.151592 (+ 1361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825976}
I20250110 02:51:31.154426 1181 consensus_queue.cc:260] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:31.154713 709 consensus_peers.cc:487] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.154860 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 228) took 4838 ms. Trace:
I20250110 02:51:31.154899 1181 ts_tablet_manager.cc:1428] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:31.154966 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:31.154937 2352 rpcz_store.cc:276] 0110 02:51:26.316406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317226 (+ 820us) service_pool.cc:225] Handling call
0110 02:51:31.154527 (+4837301us) ts_tablet_manager.cc:1145] Deleting tablet 76833ce515cf4ee9bc8fff0f626af220
0110 02:51:31.154535 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.154843 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4837156}
I20250110 02:51:31.155716 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6598311f59c6476ba339b8e57961cf67 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]). Expected version 1 got 0
I20250110 02:51:31.155983 2352 tablet_replica.cc:331] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.143013 1188 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.156148 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 9289de0630804f9aa60fdaf5f3f15f68 (table default.Table_053712a8fc234f22a1cf1caf96485c71 [id=b41f2455f36a446e96631ca1bed51d1c]) successfully deleted
W20250110 02:51:31.156365 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because the tablet was not found. No further retry: Not found: Tablet not found: 76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:31.157167 912 raft_consensus.cc:3054] T 71534cc7dd014bc1afb365de96977f8c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.157327 1188 tablet_bootstrap.cc:654] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.158459 2929 raft_consensus.cc:491] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.158564 2929 raft_consensus.cc:513] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.158810 2352 raft_consensus.cc:2238] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.159052 2352 raft_consensus.cc:2267] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.159137 2929 leader_election.cc:290] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.159874 2352 ts_tablet_manager.cc:1905] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.160161 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.160543 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:31.160820 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
W20250110 02:51:31.160851 994 consensus_peers.cc:487] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:31.161010 994 leader_election.cc:343] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.161854 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.162030 770 raft_consensus.cc:2463] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.162384 996 leader_election.cc:304] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.162983 2929 raft_consensus.cc:2798] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.163087 2929 raft_consensus.cc:491] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.163154 2929 raft_consensus.cc:3054] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.163214 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:31.163518 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:31.163702 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:31.163702 2357 ts_tablet_manager.cc:1918] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.163784 2357 log.cc:1198] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1b7ec8c7891b4a2c9fcb3cc2eeedd29a
I20250110 02:51:31.163838 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:31.164050 2357 ts_tablet_manager.cc:1939] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.164199 1450 catalog_manager.cc:5526] T 0059fbcbfb7a4f239a5947f702d21c0f P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.164562 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:31.164718 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 156) took 6132 ms. Trace:
I20250110 02:51:31.164789 2357 rpcz_store.cc:276] 0110 02:51:25.032643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.036437 (+ 3794us) service_pool.cc:225] Handling call
0110 02:51:31.119363 (+6082926us) ts_tablet_manager.cc:1145] Deleting tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
0110 02:51:31.119371 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.155395 (+ 36024us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.163622 (+ 8227us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.164702 (+ 1080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6082708}
I20250110 02:51:31.164975 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_e3e5bb662c644b5c8d10bd0f63ffe818 [id=b58434c397244de09dd22be49baeeb09]). Expected version 2 got 0
I20250110 02:51:31.165194 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a (table default.Table_e5f306c95c94407da3a3eff0c31e33a1 [id=7f4f377aed60471c96ff06042d053b80]) successfully deleted
I20250110 02:51:31.165400 2929 raft_consensus.cc:513] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.165704 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:31.165941 2929 leader_election.cc:290] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.166512 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:31.166568 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ffa250a3554a7b8f3a9c34ee7e4674" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.166700 774 raft_consensus.cc:3054] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.170531 2352 ts_tablet_manager.cc:1918] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.166008 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:31.170923 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:31.171511 1450 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
W20250110 02:51:31.172374 994 leader_election.cc:343] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.172821 1457 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_5f2cdcc0d6c94163b11c6639fdf0b9d5 [id=0692620395c14593803592e6aade6af3] alter complete (version 3)
I20250110 02:51:31.173488 2735 consensus_queue.cc:1035] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.173331 2898 consensus_queue.cc:1035] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.176209 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 157) took 5785 ms. Trace:
I20250110 02:51:31.176301 2357 rpcz_store.cc:276] 0110 02:51:25.390705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.390765 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:31.176092 (+5785327us) ts_tablet_manager.cc:1145] Deleting tablet fe860d60b3194dc5a2e7a94d39d4b8dd
0110 02:51:31.176100 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.176194 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5785141}
W20250110 02:51:31.176767 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
I20250110 02:51:31.178359 2357 tablet_replica.cc:331] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:31.178421 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 111) took 2444 ms (client timeout 1000 ms). Trace:
I20250110 02:51:31.178599 2357 raft_consensus.cc:2238] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.179448 2357 raft_consensus.cc:2267] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:31.178576 1457 rpcz_store.cc:269] 0110 02:51:28.733449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.745622 (+ 12173us) service_pool.cc:225] Handling call
0110 02:51:31.178353 (+2432731us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.884089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.884169 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.884174 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:28.884176 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.884290 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.884298 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.884303 (+ 5us) tablet.cc:682] Acquiring locks for 4 operations
0110 02:51:28.884393 (+ 90us) tablet.cc:698] Row locks acquired
0110 02:51:28.884396 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:28.884421 (+ 25us) write_op.cc:270] Start()
0110 02:51:28.884445 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488884360 usec, L: 0
0110 02:51:28.884449 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.884583 (+ 134us) log.cc:844] Serialized 1650 byte log entry
0110 02:51:28.943319 (+ 58736us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.943400 (+ 81us) write_op.cc:301] APPLY: starting
0110 02:51:28.943410 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.943461 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.943484 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.943552 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.943578 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0110 02:51:28.943584 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.943642 (+ 58us) log.cc:844] Serialized 41 byte log entry
0110 02:51:28.943686 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.943702 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:28.943706 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.952334 (+ 8628us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.952924 (+ 590us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:29.083085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:29.083238 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:29.083242 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:29.083244 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:29.083366 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:29.083372 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:29.083376 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:29.083442 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:29.083444 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:29.083506 (+ 62us) write_op.cc:270] Start()
0110 02:51:29.083526 (+ 20us) write_op.cc:276] Timestamp: P: 1736477489083481 usec, L: 0
0110 02:51:29.083529 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:29.083720 (+ 191us) log.cc:844] Serialized 356 byte log entry
0110 02:51:29.118114 (+ 34394us) op_driver.cc:464] REPLICATION: finished
0110 02:51:29.119959 (+ 1845us) write_op.cc:301] APPLY: starting
0110 02:51:29.119966 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:29.119994 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:29.120017 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:29.120048 (+ 31us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:29.120075 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:29.120081 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:29.120137 (+ 56us) log.cc:844] Serialized 23 byte log entry
0110 02:51:29.120170 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.120185 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:29.120191 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":36,"spinlock_wait_cycles":1472896,"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":354,"num_ops":4,"prepare.queue_time_us":68918,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":68239,"raft.queue_time_us":9773,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":131,"replication_time_us":58827}],["op",{"apply.queue_time_us":1816,"apply.run_cpu_time_us":280,"apply.run_wall_time_us":280,"num_ops":1,"prepare.queue_time_us":131202,"prepare.run_cpu_time_us":566,"prepare.run_wall_time_us":972,"replication_time_us":34567,"thread_start_us":122711,"threads_started":1}]]}
I20250110 02:51:31.180012 2357 ts_tablet_manager.cc:1905] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:31.180490 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 112) took 1389 ms (client timeout 1000 ms). Trace:
W20250110 02:51:31.180603 1457 rpcz_store.cc:269] 0110 02:51:29.791328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.180258 (+1388930us) service_pool.cc:212] Skipping call since client already timed out
0110 02:51:31.180442 (+ 184us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 02:51:31.180995 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 131) took 1356 ms (client timeout 1000 ms). Trace:
I20250110 02:51:31.180971 2738 consensus_queue.cc:1035] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:31.181090 1457 rpcz_store.cc:269] 0110 02:51:29.824473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.180919 (+1356446us) service_pool.cc:212] Skipping call since client already timed out
0110 02:51:31.180951 (+ 32us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250110 02:51:31.186578 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 0558a2c03f6b42a6befdc48e3af23a6d
I20250110 02:51:31.187186 1440 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 157) took 2401 ms. Trace:
I20250110 02:51:31.187551 1181 tablet_bootstrap.cc:492] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.187295 1440 rpcz_store.cc:276] 0110 02:51:28.785879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.252651 (+466772us) service_pool.cc:225] Handling call
0110 02:51:29.265508 (+ 12857us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:29.265998 (+ 490us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:29.266133 (+ 135us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:29.266198 (+ 65us) catalog_manager.cc:2762] Locking tablets
0110 02:51:30.547343 (+1281145us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:30.553069 (+ 5726us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:30.553164 (+ 95us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:31.186517 (+633353us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:31.187152 (+ 635us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:30.548045 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:30.548167 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:30.548174 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:30.548178 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:30.548308 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:30.548321 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:30.548327 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:30.548455 (+ 128us) tablet.cc:698] Row locks acquired
0110 02:51:30.548459 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:30.548494 (+ 35us) write_op.cc:270] Start()
0110 02:51:30.548525 (+ 31us) write_op.cc:276] Timestamp: P: 1736477490548470 usec, L: 0
0110 02:51:30.548530 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:30.548721 (+ 191us) log.cc:844] Serialized 759 byte log entry
0110 02:51:30.552062 (+ 3341us) op_driver.cc:464] REPLICATION: finished
0110 02:51:30.552174 (+ 112us) write_op.cc:301] APPLY: starting
0110 02:51:30.552190 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:30.552236 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:30.552258 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:30.552310 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:30.552341 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:30.552347 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:30.552406 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:30.552456 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.552472 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:30.552478 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:30.552537 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.552540 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":29013760,"child_traces":[["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":355,"apply.run_wall_time_us":354,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":800,"raft.queue_time_us":77,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":336,"replication_time_us":3505}]]}
I20250110 02:51:31.189080 1181 tablet_bootstrap.cc:654] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.189424 774 raft_consensus.cc:2463] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.189224 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 237) took 5776 ms. Trace:
I20250110 02:51:31.189848 996 leader_election.cc:304] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.189925 2224 rpcz_store.cc:276] 0110 02:51:25.412416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.412463 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:31.145221 (+5732758us) ts_tablet_manager.cc:1145] Deleting tablet 6555b0a4491046bf8f5cd4ee406c3955
0110 02:51:31.145229 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.148885 (+ 3656us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.151492 (+ 2607us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.189204 (+ 37712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5732609}
I20250110 02:51:31.190079 2898 raft_consensus.cc:2798] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.190193 2898 raft_consensus.cc:695] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.190228 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 266) took 3751 ms. Trace:
I20250110 02:51:31.190294 2224 rpcz_store.cc:276] 0110 02:51:27.439268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439303 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:31.190147 (+3750844us) ts_tablet_manager.cc:1145] Deleting tablet b43bb790e85343308a1bb1e1110bfb9c
0110 02:51:31.190155 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.190215 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3750681}
I20250110 02:51:31.190433 2898 consensus_queue.cc:237] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.190621 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 267) took 3751 ms. Trace:
I20250110 02:51:31.190691 2224 rpcz_store.cc:276] 0110 02:51:27.439330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439480 (+ 150us) service_pool.cc:225] Handling call
0110 02:51:31.190547 (+3751067us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:31.190556 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.190608 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3750938}
I20250110 02:51:31.190852 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]) successfully deleted
I20250110 02:51:31.190960 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 268) took 3751 ms. Trace:
I20250110 02:51:31.191033 2224 rpcz_store.cc:276] 0110 02:51:27.439385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439625 (+ 240us) service_pool.cc:225] Handling call
0110 02:51:31.190877 (+3751252us) ts_tablet_manager.cc:1145] Deleting tablet 5af3d61366b2432db53b18f2b7bb49d9
0110 02:51:31.190889 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.190936 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3751139}
W20250110 02:51:31.191105 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because the tablet was not found. No further retry: Not found: Tablet not found: b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:31.191273 2224 tablet_replica.cc:331] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:31.191351 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because the tablet was not found. No further retry: Not found: Tablet not found: d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:31.191481 2224 raft_consensus.cc:2238] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.191800 2224 raft_consensus.cc:2267] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:31.192219 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5af3d61366b2432db53b18f2b7bb49d9 because the tablet was not found. No further retry: Not found: Tablet not found: 5af3d61366b2432db53b18f2b7bb49d9
I20250110 02:51:31.192299 2224 ts_tablet_manager.cc:1905] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.193213 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.193549 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:31.197672 2898 raft_consensus.cc:491] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:31.197767 1449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f1d7940a1574af1804d5f5139ca7494 consensus state has no leader
I20250110 02:51:31.197824 2898 raft_consensus.cc:513] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.197934 1449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 147) took 2755 ms. Trace:
I20250110 02:51:31.198028 1449 rpcz_store.cc:276] 0110 02:51:28.442204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.732076 (+289872us) service_pool.cc:225] Handling call
0110 02:51:28.740516 (+ 8440us) spinlock_profiling.cc:243] Waited 8.19 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.740734 (+ 218us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.740915 (+ 181us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:28.741056 (+ 141us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:28.741162 (+ 106us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:28.771690 (+ 30528us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:31.197897 (+2426207us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.765550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.765627 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.765631 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:28.765633 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:28.765704 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.765711 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.765716 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.765757 (+ 41us) tablet.cc:698] Row locks acquired
0110 02:51:28.765758 (+ 1us) write_op.cc:260] PREPARE: finished
0110 02:51:28.765782 (+ 24us) write_op.cc:270] Start()
0110 02:51:28.765822 (+ 40us) write_op.cc:276] Timestamp: P: 1736477488765733 usec, L: 0
0110 02:51:28.765825 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.765955 (+ 130us) log.cc:844] Serialized 405 byte log entry
0110 02:51:28.771268 (+ 5313us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.771352 (+ 84us) write_op.cc:301] APPLY: starting
0110 02:51:28.771363 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.771411 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.771436 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.771468 (+ 32us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.771489 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.771495 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.771542 (+ 47us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.771583 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.771593 (+ 10us) write_op.cc:454] Released schema lock
0110 02:51:28.771598 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.822302 (+ 50704us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.822310 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18834816,"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":290,"apply.run_wall_time_us":291,"num_ops":1,"prepare.queue_time_us":23999,"prepare.run_cpu_time_us":527,"prepare.run_wall_time_us":56740,"raft.queue_time_us":651,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":5422}]]}
I20250110 02:51:31.199327 2898 leader_election.cc:290] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.199381 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.199579 767 raft_consensus.cc:2463] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.199945 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet eed058931194480ca1a27130bfa8ee32
I20250110 02:51:31.199972 996 leader_election.cc:304] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:31.200192 2898 raft_consensus.cc:2798] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.200248 2898 raft_consensus.cc:491] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.200294 2898 raft_consensus.cc:3054] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.200413 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 139) took 2835 ms. Trace:
I20250110 02:51:31.200515 1448 rpcz_store.cc:276] 0110 02:51:28.364714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.485734 (+121020us) service_pool.cc:225] Handling call
0110 02:51:28.677873 (+192139us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.682243 (+ 4370us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.682419 (+ 176us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:28.682478 (+ 59us) catalog_manager.cc:2762] Locking tablets
0110 02:51:28.847510 (+165032us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:29.120507 (+272997us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:29.120567 (+ 60us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:31.199862 (+2079295us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:31.200379 (+ 517us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:29.057767 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:29.080689 (+ 22922us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:29.080709 (+ 20us) write_op.cc:435] Acquired schema lock
0110 02:51:29.080713 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:29.080852 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:29.080864 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:29.080868 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:29.080964 (+ 96us) tablet.cc:698] Row locks acquired
0110 02:51:29.080966 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:29.081001 (+ 35us) write_op.cc:270] Start()
0110 02:51:29.081031 (+ 30us) write_op.cc:276] Timestamp: P: 1736477489080974 usec, L: 0
0110 02:51:29.081040 (+ 9us) op_driver.cc:348] REPLICATION: starting
0110 02:51:29.081232 (+ 192us) log.cc:844] Serialized 756 byte log entry
0110 02:51:29.117994 (+ 36762us) op_driver.cc:464] REPLICATION: finished
0110 02:51:29.119015 (+ 1021us) write_op.cc:301] APPLY: starting
0110 02:51:29.119028 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:29.119063 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:29.119092 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:29.119163 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:29.119192 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:29.119203 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:29.119263 (+ 60us) log.cc:844] Serialized 29 byte log entry
0110 02:51:29.119307 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.119321 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:29.119328 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:29.119406 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.119412 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":377602944,"child_traces":[["op",{"apply.queue_time_us":980,"apply.run_cpu_time_us":373,"apply.run_wall_time_us":373,"num_ops":2,"prepare.queue_time_us":207701,"prepare.run_cpu_time_us":864,"prepare.run_wall_time_us":24897,"replication_time_us":36928}]]}
I20250110 02:51:31.201156 1448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_1163b9629c7143d8a9aa6252b1c8b658" } modify_external_catalogs: true
I20250110 02:51:31.201328 1448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_1163b9629c7143d8a9aa6252b1c8b658" } modify_external_catalogs: true
I20250110 02:51:31.201933 1443 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet e4726347afd649afa942d74df8c013fe
W20250110 02:51:31.202030 1456 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 467f87955c4a4d27a6f0483aa109b922 consensus state has no leader
I20250110 02:51:31.202426 1443 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 140) took 2837 ms. Trace:
I20250110 02:51:31.202517 1443 rpcz_store.cc:276] 0110 02:51:28.364773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.491316 (+126543us) service_pool.cc:225] Handling call
0110 02:51:28.613427 (+122111us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.614064 (+ 637us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.614203 (+ 139us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:28.614251 (+ 48us) catalog_manager.cc:2762] Locking tablets
0110 02:51:28.846703 (+232452us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:29.120371 (+273668us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:29.120462 (+ 91us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:31.201883 (+2081421us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:31.202342 (+ 459us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:29.009316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:29.009531 (+ 215us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:29.009536 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:29.009540 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:29.009757 (+ 217us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:29.009797 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:29.009805 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:29.010006 (+ 201us) tablet.cc:698] Row locks acquired
0110 02:51:29.010008 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:29.010068 (+ 60us) write_op.cc:270] Start()
0110 02:51:29.010127 (+ 59us) write_op.cc:276] Timestamp: P: 1736477489010046 usec, L: 0
0110 02:51:29.010132 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:29.010478 (+ 346us) log.cc:844] Serialized 725 byte log entry
0110 02:51:29.117578 (+107100us) op_driver.cc:464] REPLICATION: finished
0110 02:51:29.118338 (+ 760us) write_op.cc:301] APPLY: starting
0110 02:51:29.118357 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:29.118452 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:29.118493 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:29.118586 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:29.118632 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:29.118642 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:29.118745 (+ 103us) log.cc:844] Serialized 29 byte log entry
0110 02:51:29.118827 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.118849 (+ 22us) write_op.cc:454] Released schema lock
0110 02:51:29.118857 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:29.118930 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.118936 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":276248448,"child_traces":[["op",{"apply.queue_time_us":704,"apply.run_cpu_time_us":583,"apply.run_wall_time_us":582,"num_ops":2,"prepare.queue_time_us":161972,"prepare.run_cpu_time_us":1041,"prepare.run_wall_time_us":43850,"raft.queue_time_us":31,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":782,"replication_time_us":107412}]]}
I20250110 02:51:31.203023 2898 raft_consensus.cc:513] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.203189 1443 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f015e99f53f48599ac2d7a29220bc39 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:31.177145 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178061 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.204396 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:31.204943 747 tablet_service.cc:1514] Processing DeleteTablet for tablet eed058931194480ca1a27130bfa8ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:31.204953 994 leader_election.cc:343] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.205169 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e4726347afd649afa942d74df8c013fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:31.205209 2898 leader_election.cc:290] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.178201 894 tablet_service.cc:1514] Processing DeleteTablet for tablet cf711522866d46439d7ee69aa63d7f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178294 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178395 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178750 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178826 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.178968 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.205778 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.205987 767 raft_consensus.cc:3054] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:31.206626 1444 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3aee71c51a96482fbb527576867bbaa9 consensus state has no leader
I20250110 02:51:31.209044 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet eed058931194480ca1a27130bfa8ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.209254 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet e4726347afd649afa942d74df8c013fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.194949 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.210695 767 raft_consensus.cc:2463] T e27f7ffc50494854ba34e1fbce9bee1b P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.211820 1188 tablet_bootstrap.cc:492] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.212101 1188 ts_tablet_manager.cc:1397] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20250110 02:51:31.213353 996 leader_election.cc:304] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.213259 1188 raft_consensus.cc:357] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.213681 1188 raft_consensus.cc:383] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.213809 1188 raft_consensus.cc:738] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.214155 1188 consensus_queue.cc:260] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.195748 965 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:31.198698 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.202013 893 tablet_service.cc:1514] Processing DeleteTablet for tablet eed058931194480ca1a27130bfa8ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.202661 896 tablet_service.cc:1514] Processing DeleteTablet for tablet e4726347afd649afa942d74df8c013fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.215382 2738 raft_consensus.cc:2798] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.215466 1188 ts_tablet_manager.cc:1428] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:31.215502 2738 raft_consensus.cc:695] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.215853 1188 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.215730 2738 consensus_queue.cc:237] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.216567 2352 log.cc:1198] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/302af84fd8374ffa96a7503b54849536
W20250110 02:51:31.217178 994 leader_election.cc:343] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.220463 2898 consensus_queue.cc:1035] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.222222 2352 ts_tablet_manager.cc:1939] T 302af84fd8374ffa96a7503b54849536 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.224509 1184 tablet_bootstrap.cc:492] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.224640 1184 ts_tablet_manager.cc:1397] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.004s
I20250110 02:51:31.225345 1184 raft_consensus.cc:357] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.225517 1184 raft_consensus.cc:383] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.225570 1184 raft_consensus.cc:738] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.225346 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 216) took 4909 ms. Trace:
I20250110 02:51:31.226399 1184 consensus_queue.cc:260] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.227031 2898 consensus_queue.cc:1035] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.228850 1184 ts_tablet_manager.cc:1428] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:31.229136 1184 tablet_bootstrap.cc:492] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.226133 2352 rpcz_store.cc:276] 0110 02:51:26.315404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317376 (+ 1972us) service_pool.cc:225] Handling call
0110 02:51:31.155947 (+4838571us) ts_tablet_manager.cc:1145] Deleting tablet 302af84fd8374ffa96a7503b54849536
0110 02:51:31.155954 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.168035 (+ 12081us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.170465 (+ 2430us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.225051 (+ 54586us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1,"tablet-delete.queue_time_us":4838471}
I20250110 02:51:31.230641 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 302af84fd8374ffa96a7503b54849536 (table default.Table_d9338439b43445e9b19ed1f53c53d149 [id=2c2e7dc8dfb84c8993a5119035a7c9c2]) successfully deleted
I20250110 02:51:31.230896 1181 tablet_bootstrap.cc:492] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.231086 1181 ts_tablet_manager.cc:1397] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.001s
I20250110 02:51:31.231323 2352 tablet_replica.cc:331] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.232937 1181 raft_consensus.cc:357] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.233353 1181 raft_consensus.cc:383] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.233492 1181 raft_consensus.cc:738] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.233011 2352 raft_consensus.cc:2238] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.233896 1181 consensus_queue.cc:260] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.234409 2929 consensus_queue.cc:1035] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.234676 1188 tablet_bootstrap.cc:654] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.234807 1181 ts_tablet_manager.cc:1428] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:31.235287 1181 tablet_bootstrap.cc:492] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.236315 2352 raft_consensus.cc:2267] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.239249 912 raft_consensus.cc:3054] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.244342 2908 consensus_queue.cc:1035] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.248719 1456 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 154) took 2562 ms. Trace:
I20250110 02:51:31.248849 1456 rpcz_store.cc:276] 0110 02:51:28.686240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.107594 (+421354us) service_pool.cc:225] Handling call
0110 02:51:29.108082 (+ 488us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:29.108364 (+ 282us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:29.108689 (+ 325us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:29.108760 (+ 71us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:29.253313 (+144553us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:31.248655 (+1995342us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:29.179126 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:29.179247 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:29.179255 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:29.179260 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:29.179373 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:29.179400 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:29.179405 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:29.179488 (+ 83us) tablet.cc:698] Row locks acquired
0110 02:51:29.179491 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:29.179528 (+ 37us) write_op.cc:270] Start()
0110 02:51:29.179557 (+ 29us) write_op.cc:276] Timestamp: P: 1736477489179490 usec, L: 0
0110 02:51:29.179560 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:29.242281 (+ 62721us) log.cc:844] Serialized 405 byte log entry
0110 02:51:29.245135 (+ 2854us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:29.250498 (+ 5363us) op_driver.cc:464] REPLICATION: finished
0110 02:51:29.250752 (+ 254us) write_op.cc:301] APPLY: starting
0110 02:51:29.250767 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:29.250821 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:29.250823 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:29.250875 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:29.250906 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:29.250913 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:29.250978 (+ 65us) log.cc:844] Serialized 23 byte log entry
0110 02:51:29.251034 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.251051 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:29.251060 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:29.251136 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:29.251139 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":365,"num_ops":1,"prepare.queue_time_us":69699,"prepare.run_cpu_time_us":1298,"prepare.run_wall_time_us":66155,"raft.queue_time_us":1115,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":319,"replication_time_us":70912,"spinlock_wait_cycles":143709568}]]}
I20250110 02:51:31.249836 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_88d75763367c40a2afacc85de418e9a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:31.250077 1184 tablet_bootstrap.cc:654] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:31.250461 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88d75763367c40a2afacc85de418e9a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.248744 1444 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 151) took 2678 ms. Trace:
I20250110 02:51:31.251539 1444 rpcz_store.cc:276] 0110 02:51:28.570104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.758635 (+188531us) service_pool.cc:225] Handling call
0110 02:51:28.763085 (+ 4450us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:28.763293 (+ 208us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:28.763486 (+ 193us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:28.763614 (+ 128us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:28.763678 (+ 64us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:28.844188 (+ 80510us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:31.248693 (+2404505us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:28.837362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:28.837439 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:28.837443 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:28.837446 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:28.837517 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:28.837527 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:28.837531 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:28.837575 (+ 44us) tablet.cc:698] Row locks acquired
0110 02:51:28.837577 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:28.837603 (+ 26us) write_op.cc:270] Start()
0110 02:51:28.837627 (+ 24us) write_op.cc:276] Timestamp: P: 1736477488837547 usec, L: 0
0110 02:51:28.837630 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:28.837783 (+ 153us) log.cc:844] Serialized 405 byte log entry
0110 02:51:28.843767 (+ 5984us) op_driver.cc:464] REPLICATION: finished
0110 02:51:28.843834 (+ 67us) write_op.cc:301] APPLY: starting
0110 02:51:28.843844 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:28.843880 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:28.843916 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:28.843954 (+ 38us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:28.843980 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:28.843986 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:28.844033 (+ 47us) log.cc:844] Serialized 23 byte log entry
0110 02:51:28.844080 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.844094 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:28.844099 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:28.882310 (+ 38211us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:28.882317 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":9707776,"child_traces":[["op",{"apply.queue_time_us":29,"apply.run_cpu_time_us":311,"apply.run_wall_time_us":311,"num_ops":1,"prepare.queue_time_us":73308,"prepare.run_cpu_time_us":530,"prepare.run_wall_time_us":44934,"raft.queue_time_us":700,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":4,"replication_time_us":6116}]]}
I20250110 02:51:31.252753 1444 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_3993d1a3d35d4f9cb244a585f4d0084e" } new_table_name: "default.Table_614d56e4a7544c27aa6f21776b2138f3" modify_external_catalogs: true
I20250110 02:51:31.258819 2224 ts_tablet_manager.cc:1918] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.265452 2352 ts_tablet_manager.cc:1905] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.265878 2224 log.cc:1198] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/676f5f457d9e4028bfd0b707604c42f5
W20250110 02:51:31.264552 994 consensus_peers.cc:487] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.266049 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 162) took 2331 ms. Trace:
I20250110 02:51:31.261008 1181 tablet_bootstrap.cc:654] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.266160 1456 rpcz_store.cc:276] 0110 02:51:28.934175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.249547 (+2315372us) service_pool.cc:225] Handling call
0110 02:51:31.254263 (+ 4716us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.254807 (+ 544us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.265810 (+ 11003us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.265919 (+ 109us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.265992 (+ 73us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.255285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.255382 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.255388 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.255390 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.255479 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.255492 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.255496 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.255562 (+ 66us) tablet.cc:698] Row locks acquired
0110 02:51:31.255564 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.255594 (+ 30us) write_op.cc:270] Start()
0110 02:51:31.255623 (+ 29us) write_op.cc:276] Timestamp: P: 1736477491255549 usec, L: 0
0110 02:51:31.255626 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.255808 (+ 182us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.259506 (+ 3698us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.259608 (+ 102us) write_op.cc:301] APPLY: starting
0110 02:51:31.259622 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.259679 (+ 57us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.259701 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.259818 (+ 117us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.259848 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.259856 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:31.259946 (+ 90us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.259995 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.260010 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:31.260015 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.260079 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.260082 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":467,"apply.run_wall_time_us":468,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":617,"prepare.run_wall_time_us":617,"raft.queue_time_us":58,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":3855}]]}
I20250110 02:51:31.268489 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_80f6fdd02e5349578c351409e97f722a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.269156 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80f6fdd02e5349578c351409e97f722a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.271780 2224 ts_tablet_manager.cc:1939] T 676f5f457d9e4028bfd0b707604c42f5 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
W20250110 02:51:31.285612 994 consensus_peers.cc:487] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.301054 1181 tablet_bootstrap.cc:492] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.301286 1181 ts_tablet_manager.cc:1397] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.002s
I20250110 02:51:31.302058 1181 raft_consensus.cc:357] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.302337 1181 raft_consensus.cc:383] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.302449 1181 raft_consensus.cc:738] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.302753 1181 consensus_queue.cc:260] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.307672 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 164) took 2373 ms. Trace:
I20250110 02:51:31.307802 1456 rpcz_store.cc:276] 0110 02:51:28.934381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.268164 (+2333783us) service_pool.cc:225] Handling call
0110 02:51:31.269286 (+ 1122us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.269848 (+ 562us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.288125 (+ 18277us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.307510 (+ 19385us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.307616 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.276148 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.276301 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.276309 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.276313 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.276438 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.276454 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.276462 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.276572 (+ 110us) tablet.cc:698] Row locks acquired
0110 02:51:31.276576 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.276614 (+ 38us) write_op.cc:270] Start()
0110 02:51:31.276651 (+ 37us) write_op.cc:276] Timestamp: P: 1736477491276563 usec, L: 0
0110 02:51:31.276657 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.276859 (+ 202us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.280447 (+ 3588us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.280562 (+ 115us) write_op.cc:301] APPLY: starting
0110 02:51:31.280578 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.280632 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.280668 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.280769 (+ 101us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.280804 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.280810 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:31.280877 (+ 67us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.280935 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.280953 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:31.280958 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.281039 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.281043 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":454,"num_ops":2,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":830,"prepare.run_wall_time_us":830,"raft.queue_time_us":450,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":275,"replication_time_us":3763}]]}
I20250110 02:51:31.310106 1181 ts_tablet_manager.cc:1428] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250110 02:51:31.310460 1181 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.312148 1181 tablet_bootstrap.cc:654] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:31.315670 994 consensus_peers.cc:487] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:31.315887 994 consensus_peers.cc:487] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.321458 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_e027fefdb6324ebabe8604204d2a6712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:31.322197 2357 ts_tablet_manager.cc:1918] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.326920 2357 log.cc:1198] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/6555b0a4491046bf8f5cd4ee406c3955
I20250110 02:51:31.334694 2378 raft_consensus.cc:491] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.334856 2378 raft_consensus.cc:513] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.335523 2378 leader_election.cc:290] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.338835 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.340968 853 consensus_peers.cc:487] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.347342 2738 raft_consensus.cc:491] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.347519 2738 raft_consensus.cc:513] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.348511 2898 raft_consensus.cc:491] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.348546 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.348793 2898 raft_consensus.cc:513] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:31.349159 994 leader_election.cc:343] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.351145 2898 leader_election.cc:290] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.352142 2738 leader_election.cc:290] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.354496 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.336599 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d637597047ab4738a17eca2f5409459a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.309060 1184 tablet_bootstrap.cc:492] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
W20250110 02:51:31.355087 994 leader_election.cc:343] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.338899 2952 raft_consensus.cc:491] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.355167 1184 ts_tablet_manager.cc:1397] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.005s
I20250110 02:51:31.355309 2952 raft_consensus.cc:513] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.355554 1443 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 4f015e99f53f48599ac2d7a29220bc39
I20250110 02:51:31.355948 2952 leader_election.cc:290] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.356053 1443 catalog_manager.cc:6596] ReplaceTablet: tablet 4f015e99f53f48599ac2d7a29220bc39 deleted and replaced by tablet 81138947fcab4db69bf1492de1f8560f
I20250110 02:51:31.356175 1443 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 161) took 2524 ms. Trace:
I20250110 02:51:31.356181 853 leader_election.cc:304] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:31.356271 1184 raft_consensus.cc:357] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.356452 1184 raft_consensus.cc:383] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.356511 1184 raft_consensus.cc:738] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.356662 2378 raft_consensus.cc:2743] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.356279 1443 rpcz_store.cc:276] 0110 02:51:28.832127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.203073 (+2370946us) service_pool.cc:225] Handling call
0110 02:51:31.356148 (+153075us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.345717 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.345838 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.345846 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.345850 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.345965 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.345979 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.345986 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.348220 (+ 2234us) tablet.cc:698] Row locks acquired
0110 02:51:31.348223 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.348269 (+ 46us) write_op.cc:270] Start()
0110 02:51:31.348294 (+ 25us) write_op.cc:276] Timestamp: P: 1736477491348200 usec, L: 0
0110 02:51:31.348297 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.348475 (+ 178us) log.cc:844] Serialized 622 byte log entry
0110 02:51:31.354418 (+ 5943us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.354858 (+ 440us) write_op.cc:301] APPLY: starting
0110 02:51:31.354872 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.354912 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.354933 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.355026 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.355084 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.355089 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:31.355167 (+ 78us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.355219 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.355237 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:31.355243 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.355308 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.355311 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":388,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":447,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":2840,"raft.queue_time_us":246,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":222,"replication_time_us":6094}]]}
I20250110 02:51:31.356689 1184 consensus_queue.cc:260] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.351425 764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.357101 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_7b4f8c49307a462c81559f8b98001b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:31.357270 764 raft_consensus.cc:2463] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.357486 1184 ts_tablet_manager.cc:1428] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:31.357616 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b4f8c49307a462c81559f8b98001b73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.357631 996 leader_election.cc:304] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.357758 1184 tablet_bootstrap.cc:492] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.352010 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.358063 774 raft_consensus.cc:2463] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.358383 996 leader_election.cc:304] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.358819 2898 raft_consensus.cc:2798] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.358888 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.359014 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 with delete_type TABLET_DATA_DELETED (replaced by tablet 81138947fcab4db69bf1492de1f8560f) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:31.359404 709 leader_election.cc:343] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.359196 2898 raft_consensus.cc:491] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.359787 1184 tablet_bootstrap.cc:654] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.359792 2898 raft_consensus.cc:3054] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.360091 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 with delete_type TABLET_DATA_DELETED (replaced by tablet 81138947fcab4db69bf1492de1f8560f) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:31.361248 3012 tablet.cc:1721] T ce4fe51f2e914865a2e8bbaa256b8098 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.362213 2898 raft_consensus.cc:513] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.363147 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 with delete_type TABLET_DATA_DELETED (replaced by tablet 81138947fcab4db69bf1492de1f8560f) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.363348 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:31.363799 994 leader_election.cc:343] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.363282 2898 leader_election.cc:290] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.364073 2898 raft_consensus.cc:2798] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.364148 2898 raft_consensus.cc:491] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.364753 3016 tablet.cc:1721] T ce4fe51f2e914865a2e8bbaa256b8098 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.365017 2898 raft_consensus.cc:3054] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.366169 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.366359 774 raft_consensus.cc:3054] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:31.369812 994 consensus_peers.cc:487] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.373610 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 166) took 2414 ms. Trace:
I20250110 02:51:31.373728 1443 rpcz_store.cc:276] 0110 02:51:28.959045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.356867 (+2397822us) service_pool.cc:225] Handling call
0110 02:51:31.357706 (+ 839us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.358106 (+ 400us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.373382 (+ 15276us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.373502 (+ 120us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.373578 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.363595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.363679 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.363684 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.363686 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.363771 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.363782 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.363786 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.363846 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:31.363849 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.363875 (+ 26us) write_op.cc:270] Start()
0110 02:51:31.363899 (+ 24us) write_op.cc:276] Timestamp: P: 1736477491363820 usec, L: 0
0110 02:51:31.363902 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.364036 (+ 134us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.371868 (+ 7832us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.372745 (+ 877us) write_op.cc:301] APPLY: starting
0110 02:51:31.372759 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.372796 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.372826 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.372914 (+ 88us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.372941 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.372946 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:31.373003 (+ 57us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.373048 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.373065 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:31.373069 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.373128 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.373131 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":819,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":372,"num_ops":2,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":511,"prepare.run_wall_time_us":512,"raft.queue_time_us":370,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":193,"replication_time_us":7945}]]}
I20250110 02:51:31.375672 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.376049 774 raft_consensus.cc:2463] T e2ab2376576448deb87f607b3e5a4f1c P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.376549 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_001aba654de747868b42e5c672869b6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.376869 994 consensus_peers.cc:487] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.376938 1184 tablet_bootstrap.cc:492] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.377043 1184 ts_tablet_manager.cc:1397] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
W20250110 02:51:31.377126 711 leader_election.cc:343] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:31.377169 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_001aba654de747868b42e5c672869b6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.377243 711 leader_election.cc:304] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.377452 2952 raft_consensus.cc:2743] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.377673 996 leader_election.cc:304] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.377951 1184 raft_consensus.cc:357] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.378041 2738 raft_consensus.cc:2798] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.378170 2738 raft_consensus.cc:695] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.378536 2738 consensus_queue.cc:237] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.379694 2898 raft_consensus.cc:513] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.380549 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:31.380772 2898 leader_election.cc:290] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:31.380946 994 leader_election.cc:343] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.383111 1184 raft_consensus.cc:383] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.383297 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 167) took 2421 ms. Trace:
I20250110 02:51:31.383400 1184 raft_consensus.cc:738] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.383415 1443 rpcz_store.cc:276] 0110 02:51:28.961909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.374256 (+2412347us) service_pool.cc:225] Handling call
0110 02:51:31.377410 (+ 3154us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.377883 (+ 473us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.383064 (+ 5181us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.383174 (+ 110us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.383265 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.378485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.378580 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.378585 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.378587 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.378659 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.378670 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.378674 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.378759 (+ 85us) tablet.cc:698] Row locks acquired
0110 02:51:31.378763 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.378810 (+ 47us) write_op.cc:270] Start()
0110 02:51:31.378834 (+ 24us) write_op.cc:276] Timestamp: P: 1736477491378752 usec, L: 0
0110 02:51:31.378837 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.378968 (+ 131us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.382022 (+ 3054us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.382438 (+ 416us) write_op.cc:301] APPLY: starting
0110 02:51:31.382450 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.382484 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.382504 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.382606 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.382635 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.382642 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.382696 (+ 54us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.382742 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.382759 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:31.382764 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.382855 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.382859 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":365,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":2,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":580,"raft.queue_time_us":68,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":197,"replication_time_us":3164}]]}
I20250110 02:51:31.385558 1443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_cb3b2b01aabd4f929eec83b441138ffc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.386284 1443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb3b2b01aabd4f929eec83b441138ffc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.386364 1184 consensus_queue.cc:260] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.386961 1184 ts_tablet_manager.cc:1428] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250110 02:51:31.387199 1184 tablet_bootstrap.cc:492] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.389354 1181 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.390969 1181 ts_tablet_manager.cc:1397] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.010s
I20250110 02:51:31.391718 1181 raft_consensus.cc:357] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.394951 1443 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 168) took 2366 ms. Trace:
I20250110 02:51:31.395124 1181 raft_consensus.cc:383] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.395077 1443 rpcz_store.cc:276] 0110 02:51:29.028489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.385278 (+2356789us) service_pool.cc:225] Handling call
0110 02:51:31.386491 (+ 1213us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.387600 (+ 1109us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.394686 (+ 7086us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.394690 (+ 4us) write_op.cc:454] Released schema lock
0110 02:51:31.394758 (+ 68us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.394850 (+ 92us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.394919 (+ 69us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.388817 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.388902 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.388908 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.388910 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.388982 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.388993 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.388998 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.389058 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:31.389060 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.389085 (+ 25us) write_op.cc:270] Start()
0110 02:51:31.389106 (+ 21us) write_op.cc:276] Timestamp: P: 1736477491389028 usec, L: 0
0110 02:51:31.389110 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.389245 (+ 135us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.391361 (+ 2116us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.393364 (+ 2003us) write_op.cc:301] APPLY: starting
0110 02:51:31.393379 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.393417 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.393450 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.393562 (+ 112us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.393598 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.393605 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.393661 (+ 56us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.393704 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.393720 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:31.393727 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1951,"apply.run_cpu_time_us":413,"apply.run_wall_time_us":412,"num_ops":2,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":504,"prepare.run_wall_time_us":504,"raft.queue_time_us":50,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":163,"replication_time_us":2228}]]}
I20250110 02:51:31.395542 1181 raft_consensus.cc:738] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.396292 1443 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_c7fa3a12842444fa9aa1758777787789" } new_table_name: "default.Table_83c9397ae7e84b17a964ab1d664307b2" modify_external_catalogs: true
I20250110 02:51:31.397151 1184 tablet_bootstrap.cc:654] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.396734 1181 consensus_queue.cc:260] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.381560 2357 ts_tablet_manager.cc:1939] T 6555b0a4491046bf8f5cd4ee406c3955 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.381044 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a4e7142e83493b98f71487df3f304d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.398471 774 raft_consensus.cc:3054] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.398705 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 158) took 5986 ms. Trace:
I20250110 02:51:31.398811 2357 rpcz_store.cc:276] 0110 02:51:25.412427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.412535 (+ 108us) service_pool.cc:225] Handling call
0110 02:51:31.178311 (+5765776us) ts_tablet_manager.cc:1145] Deleting tablet 6555b0a4491046bf8f5cd4ee406c3955
0110 02:51:31.178320 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.319528 (+141208us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.322121 (+ 2593us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.398684 (+ 76563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5765632}
I20250110 02:51:31.399114 2357 tablet_replica.cc:331] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:31.399253 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 6555b0a4491046bf8f5cd4ee406c3955 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]) successfully deleted
I20250110 02:51:31.399292 2357 raft_consensus.cc:2238] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.399528 2357 pending_rounds.cc:62] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:31.399772 2357 pending_rounds.cc:69] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611800450375680 op_type: NO_OP noop_request { }
I20250110 02:51:31.399813 1181 ts_tablet_manager.cc:1428] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250110 02:51:31.399924 2357 raft_consensus.cc:2883] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:31.400022 2357 raft_consensus.cc:2267] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.400059 1181 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.400485 2357 ts_tablet_manager.cc:1905] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.401434 774 raft_consensus.cc:2463] T 04a4e7142e83493b98f71487df3f304d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.401777 996 leader_election.cc:304] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.402189 2478 raft_consensus.cc:2798] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.402338 2478 raft_consensus.cc:695] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.403203 1181 tablet_bootstrap.cc:654] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:31.407878 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e027fefdb6324ebabe8604204d2a6712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.403165 2478 consensus_queue.cc:237] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.405824 2898 consensus_queue.cc:1035] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.411314 2952 raft_consensus.cc:491] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.411444 2952 raft_consensus.cc:513] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.412048 2952 leader_election.cc:290] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.412811 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.412797 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.413283 709 leader_election.cc:343] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.413035 1061 raft_consensus.cc:2463] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:31.413841 711 leader_election.cc:304] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.414057 2952 raft_consensus.cc:2798] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.414134 2952 raft_consensus.cc:491] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.414214 2952 raft_consensus.cc:3054] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.415326 914 raft_consensus.cc:3054] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.417097 2952 raft_consensus.cc:513] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.417632 2952 leader_election.cc:290] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.418321 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:31.418509 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467f87955c4a4d27a6f0483aa109b922" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:31.419021 709 leader_election.cc:343] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.418574 1061 raft_consensus.cc:3054] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.421990 1061 raft_consensus.cc:2463] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:31.422439 711 leader_election.cc:304] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.422660 2549 raft_consensus.cc:2798] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.423077 2549 raft_consensus.cc:695] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:31.423298 2549 consensus_queue.cc:237] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:31.427285 994 consensus_peers.cc:487] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.428380 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 272) took 3988 ms. Trace:
I20250110 02:51:31.428548 2224 rpcz_store.cc:276] 0110 02:51:27.439626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439754 (+ 128us) service_pool.cc:225] Handling call
0110 02:51:31.191233 (+3751479us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:31.191242 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.196458 (+ 5216us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.258682 (+ 62224us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.428335 (+169653us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151038,"tablet-delete.queue_time_us":3751395}
I20250110 02:51:31.429920 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 676f5f457d9e4028bfd0b707604c42f5 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]) successfully deleted
I20250110 02:51:31.433040 1188 tablet_bootstrap.cc:492] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.433324 1188 ts_tablet_manager.cc:1397] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.218s user 0.003s sys 0.003s
I20250110 02:51:31.434181 1188 raft_consensus.cc:357] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.434588 1188 raft_consensus.cc:383] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.434684 1188 raft_consensus.cc:738] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.434769 2600 consensus_queue.cc:1035] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.434917 1188 consensus_queue.cc:260] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.435632 1188 ts_tablet_manager.cc:1428] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:31.435927 1188 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.436923 2899 consensus_queue.cc:1035] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.438499 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 165) took 2490 ms. Trace:
I20250110 02:51:31.438643 1456 rpcz_store.cc:276] 0110 02:51:28.948081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.321161 (+2373080us) service_pool.cc:225] Handling call
0110 02:51:31.408018 (+ 86857us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.408460 (+ 442us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.427361 (+ 18901us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.427371 (+ 10us) write_op.cc:454] Released schema lock
0110 02:51:31.427529 (+ 158us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.438277 (+ 10748us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.438413 (+ 136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.409129 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.409224 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.409230 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.409234 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.409403 (+ 169us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.409417 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.409423 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.409496 (+ 73us) tablet.cc:698] Row locks acquired
0110 02:51:31.409499 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.409530 (+ 31us) write_op.cc:270] Start()
0110 02:51:31.409555 (+ 25us) write_op.cc:276] Timestamp: P: 1736477491409469 usec, L: 0
0110 02:51:31.409560 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.409717 (+ 157us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.413022 (+ 3305us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.414683 (+ 1661us) write_op.cc:301] APPLY: starting
0110 02:51:31.414695 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.414725 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.414754 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.414831 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.414856 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.414862 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:31.414913 (+ 51us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.414968 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.414982 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:31.414987 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1616,"apply.run_cpu_time_us":344,"apply.run_wall_time_us":344,"num_ops":2,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":681,"raft.queue_time_us":53,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":197,"replication_time_us":3437}]]}
I20250110 02:51:31.442123 2899 consensus_queue.cc:1035] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.445372 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:31.445590 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:31.446146 1440 catalog_manager.cc:5526] T e03d53860ab744968693cf96467c3a19 P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:31.446547 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:31.447103 1440 catalog_manager.cc:5526] T c1608d90ceb64602889b5705ba4ab55d P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:31.447474 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:31.447664 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:31.447816 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:31.447962 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:31.448127 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:31.448377 1440 catalog_manager.cc:5526] T 8bc32061409246a2b280dde4f9180a8e P 85f9597077934f1a9814d284ed06d13d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:31.448866 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:31.449141 1440 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:31.456028 2199 raft_consensus.cc:491] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.456151 2199 raft_consensus.cc:513] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.456727 2199 leader_election.cc:290] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.457433 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f9b3e74ee2431a85a22fefaacfa40e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.461191 711 leader_election.cc:343] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.466959 3016 tablet.cc:1721] T 75ca5369caf3415ea508352362409656 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.473950 2600 raft_consensus.cc:491] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.474069 2600 raft_consensus.cc:513] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.474562 2600 leader_election.cc:290] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.475953 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.476441 994 leader_election.cc:343] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.477182 2024 raft_consensus.cc:491] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.477312 2024 raft_consensus.cc:513] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.477922 2024 leader_election.cc:290] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.477962 2905 consensus_queue.cc:1035] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.479489 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.480217 994 leader_election.cc:343] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.480857 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.481076 771 raft_consensus.cc:2463] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.481506 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.481658 771 raft_consensus.cc:2463] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.485587 996 leader_election.cc:304] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.485946 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 270) took 4046 ms. Trace:
I20250110 02:51:31.485980 996 leader_election.cc:304] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.486119 2600 raft_consensus.cc:2798] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.486197 2600 raft_consensus.cc:491] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.486274 2600 raft_consensus.cc:3054] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.486301 2899 raft_consensus.cc:2798] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.486403 2899 raft_consensus.cc:491] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.486497 2899 raft_consensus.cc:3054] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.489159 2600 raft_consensus.cc:513] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.489339 2899 raft_consensus.cc:513] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.489674 2600 leader_election.cc:290] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.489974 2899 leader_election.cc:290] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.490272 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:31.490672 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:31.490726 2905 raft_consensus.cc:491] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.490909 2905 raft_consensus.cc:513] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.491470 2905 leader_election.cc:290] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.491648 3014 tablet.cc:1721] T ce4fe51f2e914865a2e8bbaa256b8098 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:31.491871 994 leader_election.cc:343] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.492295 2224 rpcz_store.cc:276] 0110 02:51:27.439508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439930 (+ 422us) service_pool.cc:225] Handling call
0110 02:51:31.485817 (+4045887us) ts_tablet_manager.cc:1145] Deleting tablet d4f1d916a5ea48708c2c1a61c92fe137
0110 02:51:31.485830 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.485900 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4045783}
I20250110 02:51:31.492396 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.492439 994 leader_election.cc:343] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:31.492816 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because the tablet was not found. No further retry: Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137
W20250110 02:51:31.492838 994 leader_election.cc:343] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.493266 2224 tablet_replica.cc:331] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:31.493546 2224 raft_consensus.cc:2238] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.493796 2224 raft_consensus.cc:2267] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.494428 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13683fbe9d654575b28108de8b77f943" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.494490 2224 ts_tablet_manager.cc:1905] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.494581 767 raft_consensus.cc:3054] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.495751 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f9b3e74ee2431a85a22fefaacfa40e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.496229 709 leader_election.cc:343] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.496315 709 leader_election.cc:304] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.496881 767 raft_consensus.cc:2463] T 13683fbe9d654575b28108de8b77f943 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.497085 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 75bd10c31870438cbc952e137d92eee7 (DEFAULT_TABLE table=default.Table_becbebb399f04a568ef57354161f2bb7 [id=d50443941c184939b7d13139b0e6c503]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:31.497145 994 consensus_peers.cc:487] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.497440 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bd10c31870438cbc952e137d92eee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.497532 893 tablet_service.cc:1467] Processing CreateTablet for tablet 75bd10c31870438cbc952e137d92eee7 (DEFAULT_TABLE table=default.Table_becbebb399f04a568ef57354161f2bb7 [id=d50443941c184939b7d13139b0e6c503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.497807 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bd10c31870438cbc952e137d92eee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.503671 2952 raft_consensus.cc:2743] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.504264 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.504410 775 raft_consensus.cc:3054] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.504763 765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.504899 765 raft_consensus.cc:2463] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.505642 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet e804fe33c8ae489bbc4d3765c64b59bd from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:31.505915 996 leader_election.cc:304] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.506173 2905 raft_consensus.cc:2798] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.506273 2905 raft_consensus.cc:695] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.506505 2905 consensus_queue.cc:237] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.506717 775 raft_consensus.cc:2463] T 9395aa9b39d04ac9b6cecf106bc91499 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.507683 996 leader_election.cc:304] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.508296 2024 raft_consensus.cc:2798] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.508395 2024 raft_consensus.cc:491] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.508512 2024 raft_consensus.cc:3054] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.511788 2024 raft_consensus.cc:513] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.512434 2024 leader_election.cc:290] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.512655 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:31.513159 994 leader_election.cc:343] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.515545 2952 raft_consensus.cc:491] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.515682 2952 raft_consensus.cc:513] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.516729 2952 leader_election.cc:290] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.515563 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.516966 775 raft_consensus.cc:3054] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.518863 3014 tablet.cc:1721] T 75ca5369caf3415ea508352362409656 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.519273 775 raft_consensus.cc:2463] T eadb829d657346eabcb7d2aa34b33fe2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.519737 996 leader_election.cc:304] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.520202 2600 raft_consensus.cc:2798] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.520365 2600 raft_consensus.cc:695] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.520650 2600 consensus_queue.cc:237] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.522040 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.522568 709 leader_election.cc:343] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.523038 756 tablet_service.cc:1467] Processing CreateTablet for tablet 75bd10c31870438cbc952e137d92eee7 (DEFAULT_TABLE table=default.Table_becbebb399f04a568ef57354161f2bb7 [id=d50443941c184939b7d13139b0e6c503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.523458 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bd10c31870438cbc952e137d92eee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.524308 996 leader_election.cc:304] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.524727 2899 raft_consensus.cc:2798] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.524793 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.524904 2899 raft_consensus.cc:695] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.525424 2899 consensus_queue.cc:237] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:31.526886 711 leader_election.cc:343] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.526993 711 leader_election.cc:304] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.527300 2549 raft_consensus.cc:2743] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.531847 2357 ts_tablet_manager.cc:1918] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.532251 3012 tablet.cc:1721] T 6c4bd5e8317a4801a8134d58c23b7bee P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.539229 893 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0 (DEFAULT_TABLE table=default.Table_4f07307b0947487b9111c6249489de28 [id=752b5ab9d31e43d8ad28ec4d6758b2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.539561 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0 (DEFAULT_TABLE table=default.Table_4f07307b0947487b9111c6249489de28 [id=752b5ab9d31e43d8ad28ec4d6758b2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.539671 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.540012 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.544360 3016 tablet.cc:1721] T 6c4bd5e8317a4801a8134d58c23b7bee P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.558912 1188 tablet_bootstrap.cc:654] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.563218 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet e804fe33c8ae489bbc4d3765c64b59bd
W20250110 02:51:31.571380 994 consensus_peers.cc:487] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d761ef0d1e5542b1979c29bb2d579489. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.572718 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2fd9b07cb5604fed8b3eb1b576353940) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.579105 1456 catalog_manager.cc:6596] ReplaceTablet: tablet e804fe33c8ae489bbc4d3765c64b59bd deleted and replaced by tablet 2fd9b07cb5604fed8b3eb1b576353940
I20250110 02:51:31.579356 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 170) took 2515 ms. Trace:
I20250110 02:51:31.579540 1456 rpcz_store.cc:276] 0110 02:51:29.063536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.505529 (+2441993us) service_pool.cc:225] Handling call
0110 02:51:31.562860 (+ 57331us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.562870 (+ 10us) write_op.cc:454] Released schema lock
0110 02:51:31.579310 (+ 16440us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.507073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.507198 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.507206 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.507210 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.507365 (+ 155us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.507393 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.507399 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.507489 (+ 90us) tablet.cc:698] Row locks acquired
0110 02:51:31.507493 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.507526 (+ 33us) write_op.cc:270] Start()
0110 02:51:31.507563 (+ 37us) write_op.cc:276] Timestamp: P: 1736477491507507 usec, L: 0
0110 02:51:31.507568 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.507728 (+ 160us) log.cc:844] Serialized 656 byte log entry
0110 02:51:31.515386 (+ 7658us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.517658 (+ 2272us) write_op.cc:301] APPLY: starting
0110 02:51:31.517693 (+ 35us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.517736 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.517765 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.517876 (+ 111us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.517941 (+ 65us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.517968 (+ 27us) write_op.cc:312] APPLY: finished
0110 02:51:31.518043 (+ 75us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.518108 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.518140 (+ 32us) write_op.cc:454] Released schema lock
0110 02:51:31.518148 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2213,"apply.run_cpu_time_us":552,"apply.run_wall_time_us":552,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":750,"prepare.run_wall_time_us":750,"raft.queue_time_us":65,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":243,"replication_time_us":7785}]]}
I20250110 02:51:31.557587 756 tablet_service.cc:1467] Processing CreateTablet for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0 (DEFAULT_TABLE table=default.Table_4f07307b0947487b9111c6249489de28 [id=752b5ab9d31e43d8ad28ec4d6758b2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.583940 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9fb78ffee9a44c6b3f263de6fdaf8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.573454 747 tablet_service.cc:1514] Processing DeleteTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2fd9b07cb5604fed8b3eb1b576353940) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:31.576334 3014 tablet.cc:1721] T 6c4bd5e8317a4801a8134d58c23b7bee P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.592037 2357 log.cc:1198] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/dba0cb7970f1453dac256b255ed7cac0
I20250110 02:51:31.592440 2357 ts_tablet_manager.cc:1939] T dba0cb7970f1453dac256b255ed7cac0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.593181 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 171) took 4114 ms. Trace:
I20250110 02:51:31.593317 2357 rpcz_store.cc:276] 0110 02:51:27.479065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.479133 (+ 68us) service_pool.cc:225] Handling call
0110 02:51:31.399072 (+3919939us) ts_tablet_manager.cc:1145] Deleting tablet dba0cb7970f1453dac256b255ed7cac0
0110 02:51:31.399081 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.405034 (+ 5953us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.531772 (+126738us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.593164 (+ 61392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3913812}
I20250110 02:51:31.593791 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet dba0cb7970f1453dac256b255ed7cac0 (table default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]) successfully deleted
I20250110 02:51:31.571003 896 tablet_service.cc:1514] Processing DeleteTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2fd9b07cb5604fed8b3eb1b576353940) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.571869 2352 ts_tablet_manager.cc:1918] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.595260 2352 log.cc:1198] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/3510e86314144a2ca012aacf3d78eacc
I20250110 02:51:31.595674 2352 ts_tablet_manager.cc:1939] T 3510e86314144a2ca012aacf3d78eacc P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.595997 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 218) took 5280 ms. Trace:
I20250110 02:51:31.596134 2352 rpcz_store.cc:276] 0110 02:51:26.315555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317498 (+ 1943us) service_pool.cc:225] Handling call
0110 02:51:31.230884 (+4913386us) ts_tablet_manager.cc:1145] Deleting tablet 3510e86314144a2ca012aacf3d78eacc
0110 02:51:31.230897 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.431327 (+200430us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.571770 (+140443us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.595981 (+ 24211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4912909}
I20250110 02:51:31.596657 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 3510e86314144a2ca012aacf3d78eacc (table default.Table_108282f5803c488098a01d20b1e4f31a [id=86d7982c999f46f5966d851234e922b0]) successfully deleted
I20250110 02:51:31.597189 2352 tablet_replica.cc:331] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.597467 2352 raft_consensus.cc:2238] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.597848 2352 pending_rounds.cc:62] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Trying to abort 1 pending ops.
I20250110 02:51:31.597939 2352 pending_rounds.cc:69] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611780198330368 op_type: NO_OP noop_request { }
I20250110 02:51:31.598090 2352 raft_consensus.cc:2883] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:31.598228 2352 raft_consensus.cc:2267] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.598871 2352 ts_tablet_manager.cc:1905] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.601236 1184 tablet_bootstrap.cc:492] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.601382 1184 ts_tablet_manager.cc:1397] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.214s user 0.006s sys 0.000s
I20250110 02:51:31.602365 3012 tablet.cc:1721] T 815767a4c7624852bfedab9fa6cb5f44 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.603062 3016 tablet.cc:1721] T 815767a4c7624852bfedab9fa6cb5f44 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.611471 3012 tablet.cc:1721] T 75ca5369caf3415ea508352362409656 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:31.619693 853 consensus_peers.cc:487] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.621431 3012 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.622488 2738 consensus_queue.cc:1035] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.627414 1181 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.627885 1181 ts_tablet_manager.cc:1397] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.228s user 0.007s sys 0.015s
I20250110 02:51:31.628618 1181 raft_consensus.cc:357] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.629043 1181 raft_consensus.cc:383] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.629200 1181 raft_consensus.cc:738] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.629427 1181 consensus_queue.cc:260] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.629940 1181 ts_tablet_manager.cc:1428] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:31.630366 1181 tablet_bootstrap.cc:492] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.634032 1181 tablet_bootstrap.cc:654] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.632614 3055 tablet.cc:1721] T 00ebf8debb9d4394839d1c1cb97820a5 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:31.635895 3016 tablet.cc:1721] T 00ebf8debb9d4394839d1c1cb97820a5 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:31.636329 3056 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.641439 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_2496754973df49aca9cc870a77615db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.642235 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2496754973df49aca9cc870a77615db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.645141 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 172) took 4165 ms. Trace:
I20250110 02:51:31.646114 2357 rpcz_store.cc:276] 0110 02:51:27.479343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.479392 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:31.645030 (+4165638us) ts_tablet_manager.cc:1145] Deleting tablet 9289de0630804f9aa60fdaf5f3f15f68
0110 02:51:31.645042 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.645119 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4159712}
W20250110 02:51:31.647229 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because the tablet was not found. No further retry: Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:31.647339 2357 tablet_replica.cc:331] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:31.648152 2357 raft_consensus.cc:2238] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.648994 2357 raft_consensus.cc:2267] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:31.650125 994 consensus_peers.cc:487] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.651311 2357 ts_tablet_manager.cc:1905] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.655009 2224 ts_tablet_manager.cc:1918] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.656546 2224 log.cc:1198] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/24c4042c38de4969889f4821176fc7bb
I20250110 02:51:31.666147 3014 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.666867 2224 ts_tablet_manager.cc:1939] T 24c4042c38de4969889f4821176fc7bb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.670490 1188 tablet_bootstrap.cc:492] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.670730 1188 ts_tablet_manager.cc:1397] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.235s user 0.007s sys 0.000s
I20250110 02:51:31.670749 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 276) took 4230 ms. Trace:
I20250110 02:51:31.672443 1184 raft_consensus.cc:357] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.671655 1188 raft_consensus.cc:357] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.672855 1188 raft_consensus.cc:383] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.673105 1188 raft_consensus.cc:738] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.670857 2224 rpcz_store.cc:276] 0110 02:51:27.439853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.440042 (+ 189us) service_pool.cc:225] Handling call
0110 02:51:31.493185 (+4053143us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:31.493197 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.567909 (+ 74712us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.654902 (+ 86993us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.670715 (+ 15813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4053042}
I20250110 02:51:31.673501 1188 consensus_queue.cc:260] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.673929 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 269) took 4234 ms. Trace:
I20250110 02:51:31.674057 2224 rpcz_store.cc:276] 0110 02:51:27.439454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.439968 (+ 514us) service_pool.cc:225] Handling call
0110 02:51:31.673816 (+4233848us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:31.673825 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.673887 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4233671}
I20250110 02:51:31.674309 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 24c4042c38de4969889f4821176fc7bb (table default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]) successfully deleted
I20250110 02:51:31.674574 2224 tablet_replica.cc:331] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:31.674752 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because the tablet was not found. No further retry: Not found: Tablet not found: 3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:31.675143 1188 ts_tablet_manager.cc:1428] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250110 02:51:31.675666 1181 tablet_bootstrap.cc:492] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.675860 1181 ts_tablet_manager.cc:1397] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250110 02:51:31.675901 1184 raft_consensus.cc:383] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.676056 1184 raft_consensus.cc:738] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.676306 1184 consensus_queue.cc:260] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.676855 1184 ts_tablet_manager.cc:1428] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20250110 02:51:31.676923 1181 raft_consensus.cc:357] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.677160 1181 raft_consensus.cc:383] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.677277 1181 raft_consensus.cc:738] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.677520 1181 consensus_queue.cc:260] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.678151 1181 ts_tablet_manager.cc:1428] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:31.678488 1181 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.679986 1181 tablet_bootstrap.cc:654] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.681311 2478 consensus_queue.cc:1035] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.681537 2905 consensus_queue.cc:1035] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.681809 2224 raft_consensus.cc:2238] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.682571 2224 raft_consensus.cc:2267] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.683166 2224 ts_tablet_manager.cc:1905] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.683583 3014 tablet.cc:1721] T 00ebf8debb9d4394839d1c1cb97820a5 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:31.684413 3037 tablet.cc:1721] T 815767a4c7624852bfedab9fa6cb5f44 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.677199 1184 tablet_bootstrap.cc:492] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.686808 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 171) took 2375 ms. Trace:
I20250110 02:51:31.687047 1456 rpcz_store.cc:276] 0110 02:51:29.311479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.640243 (+2328764us) service_pool.cc:225] Handling call
0110 02:51:31.642442 (+ 2199us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.643139 (+ 697us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.686438 (+ 43299us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.686448 (+ 10us) write_op.cc:454] Released schema lock
0110 02:51:31.686533 (+ 85us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.686647 (+ 114us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.686756 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.643628 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.643705 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.643711 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.643713 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.643807 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.643819 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.643823 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.643886 (+ 63us) tablet.cc:698] Row locks acquired
0110 02:51:31.643888 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.643917 (+ 29us) write_op.cc:270] Start()
0110 02:51:31.643945 (+ 28us) write_op.cc:276] Timestamp: P: 1736477491643886 usec, L: 0
0110 02:51:31.643947 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.644141 (+ 194us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.647897 (+ 3756us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.648016 (+ 119us) write_op.cc:301] APPLY: starting
0110 02:51:31.648029 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.648077 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.648100 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.648216 (+ 116us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.648255 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.648266 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:31.648361 (+ 95us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.648422 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.648444 (+ 22us) write_op.cc:454] Released schema lock
0110 02:51:31.648453 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":509,"apply.run_wall_time_us":508,"num_ops":2,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":605,"prepare.run_wall_time_us":605,"raft.queue_time_us":378,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":244,"replication_time_us":3921}]]}
I20250110 02:51:31.691583 1188 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.692813 1108 heartbeater.cc:346] Connected to a master server at 127.0.104.61:44449
I20250110 02:51:31.694224 1188 tablet_bootstrap.cc:654] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.695799 2357 ts_tablet_manager.cc:1918] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.698479 2357 log.cc:1198] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c5601ca24b8f428e8a51cbeec3664901
I20250110 02:51:31.698952 2357 ts_tablet_manager.cc:1939] T c5601ca24b8f428e8a51cbeec3664901 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.701609 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_df7558651391471ba93ced530307d670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.702306 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df7558651391471ba93ced530307d670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.701246 1184 tablet_bootstrap.cc:654] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.712455 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 187) took 4227 ms. Trace:
I20250110 02:51:31.712617 2357 rpcz_store.cc:276] 0110 02:51:27.485140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.485465 (+ 325us) service_pool.cc:225] Handling call
0110 02:51:31.647289 (+4161824us) ts_tablet_manager.cc:1145] Deleting tablet c5601ca24b8f428e8a51cbeec3664901
0110 02:51:31.647299 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.654153 (+ 6854us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.695706 (+ 41553us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.712431 (+ 16725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4161600}
I20250110 02:51:31.713253 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c5601ca24b8f428e8a51cbeec3664901 (table default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]) successfully deleted
I20250110 02:51:31.713325 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 172) took 2259 ms. Trace:
I20250110 02:51:31.713721 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 173) took 4233 ms. Trace:
I20250110 02:51:31.713852 2357 rpcz_store.cc:276] 0110 02:51:27.479763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.479809 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:31.713603 (+4233794us) ts_tablet_manager.cc:1145] Deleting tablet 940445840db3441598eba57d3ccb8c9c
0110 02:51:31.713616 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.713702 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4227783}
I20250110 02:51:31.713469 1456 rpcz_store.cc:276] 0110 02:51:29.454002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.701372 (+2247370us) service_pool.cc:225] Handling call
0110 02:51:31.702480 (+ 1108us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.702901 (+ 421us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.712673 (+ 9772us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.712682 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:31.712789 (+ 107us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.713145 (+ 356us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.713271 (+ 126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.703490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.703585 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.703592 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:31.703594 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.703674 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.703686 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.703690 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.703761 (+ 71us) tablet.cc:698] Row locks acquired
0110 02:51:31.703764 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.703794 (+ 30us) write_op.cc:270] Start()
0110 02:51:31.703824 (+ 30us) write_op.cc:276] Timestamp: P: 1736477491703757 usec, L: 0
0110 02:51:31.703828 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.703985 (+ 157us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.707257 (+ 3272us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.707352 (+ 95us) write_op.cc:301] APPLY: starting
0110 02:51:31.707366 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.707401 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.707422 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.707507 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.707534 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.707541 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.707599 (+ 58us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.707645 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.707663 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:31.707669 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":358,"apply.run_wall_time_us":357,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":579,"prepare.run_wall_time_us":579,"raft.queue_time_us":391,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":195,"replication_time_us":3400}]]}
I20250110 02:51:31.714802 1181 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.715006 1181 ts_tablet_manager.cc:1397] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250110 02:51:31.715873 1181 raft_consensus.cc:357] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.716171 1181 raft_consensus.cc:383] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.716274 1181 raft_consensus.cc:738] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.716508 1181 consensus_queue.cc:260] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.717059 1181 ts_tablet_manager.cc:1428] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:31.717628 1181 tablet_bootstrap.cc:492] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.718899 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 3510e86314144a2ca012aacf3d78eacc from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:31.719676 994 consensus_peers.cc:487] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:31.722432 1434 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet deaa3527aa074e6ba94960c672aee028 consensus state has no leader
W20250110 02:51:31.722811 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because the tablet was not found. No further retry: Not found: Tablet not found: 940445840db3441598eba57d3ccb8c9c
I20250110 02:51:31.723062 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 173) took 2183 ms. Trace:
I20250110 02:51:31.723177 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 174) took 4242 ms. Trace:
I20250110 02:51:31.723256 2357 rpcz_store.cc:276] 0110 02:51:27.480222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.480262 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:31.723093 (+4242831us) ts_tablet_manager.cc:1145] Deleting tablet e015a41a03f84c91a45e46e4b7b5e7a9
0110 02:51:31.723101 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.723161 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4237028}
I20250110 02:51:31.723186 1456 rpcz_store.cc:276] 0110 02:51:29.536287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.718828 (+2182541us) service_pool.cc:225] Handling call
0110 02:51:31.719743 (+ 915us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250110 02:51:31.723654 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because the tablet was not found. No further retry: Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:31.724054 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 176) took 4243 ms. Trace:
I20250110 02:51:31.724969 2352 ts_tablet_manager.cc:1918] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.723807 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_8cc1ce12b9c447f5a96de708a52c56ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:31.727116 2357 rpcz_store.cc:276] 0110 02:51:27.480869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.480912 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:31.723981 (+4243069us) ts_tablet_manager.cc:1145] Deleting tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
0110 02:51:31.723991 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.724041 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4237854}
W20250110 02:51:31.725950 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cc1ce12b9c447f5a96de708a52c56ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.728327 2352 log.cc:1198] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:31.728724 2352 ts_tablet_manager.cc:1939] T 8f15078fc22f4b32a48c7190b36e2192 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.730171 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:31.731187 1181 tablet_bootstrap.cc:654] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.730934 1188 tablet_bootstrap.cc:492] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:31.731611 1188 ts_tablet_manager.cc:1397] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250110 02:51:31.732261 1188 raft_consensus.cc:357] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.732473 1188 raft_consensus.cc:383] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.732570 1188 raft_consensus.cc:738] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.732820 1188 consensus_queue.cc:260] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.733335 1188 ts_tablet_manager.cc:1428] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:31.735610 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
W20250110 02:51:31.738246 853 consensus_peers.cc:487] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.739089 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:31.739347 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:31.739814 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:31.743445 1457 catalog_manager.cc:5526] T d3858638b6fc48d0bb8b8f97370a791d P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.743948 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:31.744230 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:31.744503 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:31.744776 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:31.744961 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:31.738879 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 174) took 2086 ms. Trace:
I20250110 02:51:31.745127 1456 rpcz_store.cc:276] 0110 02:51:29.652572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.723591 (+2071019us) service_pool.cc:225] Handling call
0110 02:51:31.727450 (+ 3859us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.727860 (+ 410us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.738627 (+ 10767us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.738750 (+ 123us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.738827 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.734070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.734176 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.734182 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.734184 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.734265 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.734275 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.734280 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.734349 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:31.734352 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.734398 (+ 46us) write_op.cc:270] Start()
0110 02:51:31.734429 (+ 31us) write_op.cc:276] Timestamp: P: 1736477491734357 usec, L: 0
0110 02:51:31.734432 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.734593 (+ 161us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.737165 (+ 2572us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.737257 (+ 92us) write_op.cc:301] APPLY: starting
0110 02:51:31.737269 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.737301 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.737330 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.737424 (+ 94us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.737452 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.737458 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:31.737513 (+ 55us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.737574 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.737590 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:31.737596 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.737655 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.737660 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":388,"num_ops":2,"prepare.queue_time_us":5783,"prepare.run_cpu_time_us":605,"prepare.run_wall_time_us":605,"raft.queue_time_us":353,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":203,"replication_time_us":2686}]]}
I20250110 02:51:31.746333 1456 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_74af5f41ea724f13baa721db4063a9d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:31.746971 1456 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74af5f41ea724f13baa721db4063a9d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:31.739682 2549 raft_consensus.cc:491] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.747296 2549 raft_consensus.cc:513] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.747958 2549 leader_election.cc:290] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.748485 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.748965 709 leader_election.cc:343] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.750248 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:31.750766 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:31.751068 1457 catalog_manager.cc:5526] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:31.751508 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:31.751878 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:31.752825 1456 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 175) took 2085 ms. Trace:
I20250110 02:51:31.752936 1456 rpcz_store.cc:276] 0110 02:51:29.667124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.746133 (+2079009us) service_pool.cc:225] Handling call
0110 02:51:31.747099 (+ 966us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:31.748343 (+ 1244us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:31.752633 (+ 4290us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:31.752718 (+ 85us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:31.752784 (+ 66us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.748998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.749082 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.749088 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.749090 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.749174 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.749202 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.749206 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.749322 (+ 116us) tablet.cc:698] Row locks acquired
0110 02:51:31.749325 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.749359 (+ 34us) write_op.cc:270] Start()
0110 02:51:31.749387 (+ 28us) write_op.cc:276] Timestamp: P: 1736477491749317 usec, L: 0
0110 02:51:31.749390 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.749539 (+ 149us) log.cc:844] Serialized 433 byte log entry
0110 02:51:31.752106 (+ 2567us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.752211 (+ 105us) write_op.cc:301] APPLY: starting
0110 02:51:31.752223 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.752260 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.752280 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.752359 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.752385 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.752392 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.752453 (+ 61us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.752506 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.752521 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:31.752527 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.752580 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.752584 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":363,"apply.run_wall_time_us":363,"num_ops":2,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":632,"raft.queue_time_us":31,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":260,"replication_time_us":2691}]]}
I20250110 02:51:31.753896 1184 tablet_bootstrap.cc:492] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.754045 1184 ts_tablet_manager.cc:1397] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.077s user 0.000s sys 0.004s
I20250110 02:51:31.754418 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 6598311f59c6476ba339b8e57961cf67 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:31.752444 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:31.754812 1184 raft_consensus.cc:357] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.755028 1184 raft_consensus.cc:383] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.755129 1184 raft_consensus.cc:738] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.755429 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3aee71c51a96482fbb527576867bbaa9 (table default.Table_a99fe640359046aa8ff288db7a687676 [id=c9a367ed70c742a18d7dc76e72b37300]). Expected version 1 got 0
I20250110 02:51:31.755388 1184 consensus_queue.cc:260] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.756062 1184 ts_tablet_manager.cc:1428] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:31.755954 1457 catalog_manager.cc:5526] T 84adc1fd3f76497b98a0fcbc52a3120a P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.756354 1184 tablet_bootstrap.cc:492] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.756431 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:31.757174 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:31.757745 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:31.758015 2424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35486 (request call id 0) took 6585 ms. Trace:
I20250110 02:51:31.758420 3059 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d8d40556dc348e8aa8255d116251a10)
I20250110 02:51:31.758764 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8d40556dc348e8aa8255d116251a10. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:31.758695 3059 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:31.759068 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:31.759099 1457 catalog_manager.cc:5526] T 71534cc7dd014bc1afb365de96977f8c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.759764 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:31.760114 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:31.760687 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:31.761447 1457 catalog_manager.cc:5526] T 39cfc508c1b84299a2f16d307bb25354 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.761776 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:31.762166 1457 catalog_manager.cc:5526] T b25bbf8aa71e478587093627c715310f P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.762768 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:31.762650 3066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:31.762944 3066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [sys.catalog]: This master's current role is: FOLLOWER
I20250110 02:51:31.763207 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:31.763530 3059 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:31.763831 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4f1d7940a1574af1804d5f5139ca7494 (table default.Table_b4223b31a5974abb9725bceb8026a8ae [id=4e2fb95ceb8c43fc920b1cd9e73557df]). Expected version 1 got 0
I20250110 02:51:31.770668 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:31.771983 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 6598311f59c6476ba339b8e57961cf67 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]). Expected version 1 got 0
I20250110 02:51:31.772737 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:31.773253 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:31.773763 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:31.774264 1457 catalog_manager.cc:5526] T b010e9c549214937889fa281542e752e P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.775471 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
W20250110 02:51:31.776113 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because the tablet was not found. No further retry: Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:31.776033 1457 catalog_manager.cc:5526] T ab302fedec4f4c03989b5d3932dc8d4c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:31.776456 2549 raft_consensus.cc:491] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.776584 2549 raft_consensus.cc:513] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.776925 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:31.777213 2549 leader_election.cc:290] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.777828 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.777140 1457 catalog_manager.cc:5526] T 8c7fbdab52014dddbf63cfef69b09b7b P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.778443 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
W20250110 02:51:31.778494 709 leader_election.cc:343] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.778936 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 177) took 4297 ms. Trace:
I20250110 02:51:31.779058 2357 rpcz_store.cc:276] 0110 02:51:27.481003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.481088 (+ 85us) service_pool.cc:225] Handling call
0110 02:51:31.778770 (+4297682us) ts_tablet_manager.cc:1145] Deleting tablet 6e0b2ec1d5814144bc58261e6c6e8d35
0110 02:51:31.778780 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.778920 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4292581}
I20250110 02:51:31.778924 1457 catalog_manager.cc:5526] T c52f74fe375440a28139c27f3f868970 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
W20250110 02:51:31.779528 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because the tablet was not found. No further retry: Not found: Tablet not found: 6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:31.779758 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 178) took 4298 ms. Trace:
I20250110 02:51:31.779868 2357 rpcz_store.cc:276] 0110 02:51:27.481108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.481249 (+ 141us) service_pool.cc:225] Handling call
0110 02:51:31.779678 (+4298429us) ts_tablet_manager.cc:1145] Deleting tablet 0d5ddbc6b02742609bc2ca84c863a9d0
0110 02:51:31.779688 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.779744 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4293478}
I20250110 02:51:31.780301 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:31.780901 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:31.783095 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 179) took 4301 ms. Trace:
I20250110 02:51:31.783594 2357 rpcz_store.cc:276] 0110 02:51:27.481737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.481782 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:31.783026 (+4301244us) ts_tablet_manager.cc:1145] Deleting tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
0110 02:51:31.783034 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.783078 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4296796}
I20250110 02:51:31.784371 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 180) took 4302 ms. Trace:
W20250110 02:51:31.782778 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 0d5ddbc6b02742609bc2ca84c863a9d0 because the tablet was not found. No further retry: Not found: Tablet not found: 0d5ddbc6b02742609bc2ca84c863a9d0
W20250110 02:51:31.784651 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because the tablet was not found. No further retry: Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:31.785439 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:31.785553 2357 rpcz_store.cc:276] 0110 02:51:27.482139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.482184 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:31.784307 (+4302123us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:31.784315 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.784357 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4298031}
W20250110 02:51:31.786283 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because the tablet was not found. No further retry: Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:31.786672 2357 tablet_replica.cc:331] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:31.786710 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:31.747123 2224 ts_tablet_manager.cc:1918] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.787396 2224 log.cc:1198] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d00698a0a9ee4e5da69a745f4a3aa8b4
I20250110 02:51:31.787735 2224 ts_tablet_manager.cc:1939] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.787185 1457 catalog_manager.cc:5526] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:31.788199 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:31.788543 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 271) took 4348 ms. Trace:
I20250110 02:51:31.788400 1457 catalog_manager.cc:5526] T 9e35fe84c1e04d3eb295a6d8d5d40ea4 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } }
I20250110 02:51:31.789103 1457 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:31.789547 2224 rpcz_store.cc:276] 0110 02:51:27.439573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.440166 (+ 593us) service_pool.cc:225] Handling call
0110 02:51:31.674518 (+4234352us) ts_tablet_manager.cc:1145] Deleting tablet d00698a0a9ee4e5da69a745f4a3aa8b4
0110 02:51:31.674527 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.699173 (+ 24646us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.746996 (+ 47823us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.788507 (+ 41511us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":558,"tablet-delete.queue_time_us":4234205}
I20250110 02:51:31.748338 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.789939 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 273) took 4350 ms. Trace:
I20250110 02:51:31.790028 2224 rpcz_store.cc:276] 0110 02:51:27.439683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.440293 (+ 610us) service_pool.cc:225] Handling call
0110 02:51:31.789848 (+4349555us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:31.789856 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.789907 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4349436}
I20250110 02:51:31.790298 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 275) took 4350 ms. Trace:
I20250110 02:51:31.790397 2224 rpcz_store.cc:276] 0110 02:51:27.439799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.440431 (+ 632us) service_pool.cc:225] Handling call
0110 02:51:31.790218 (+4349787us) ts_tablet_manager.cc:1145] Deleting tablet 1390f867a3a8436092d1920daff4c5b0
0110 02:51:31.790225 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.790267 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4349679}
I20250110 02:51:31.791216 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]) successfully deleted
W20250110 02:51:31.791455 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because the tablet was not found. No further retry: Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:31.797655 2357 raft_consensus.cc:2238] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.797839 2357 raft_consensus.cc:2267] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.787050 1184 tablet_bootstrap.cc:654] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:31.794493 711 leader_election.cc:343] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.798496 711 leader_election.cc:304] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.798909 2549 raft_consensus.cc:2743] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.801864 3074 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250110 02:51:31.810117 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 6598311f59c6476ba339b8e57961cf67
I20250110 02:51:31.810993 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6598311f59c6476ba339b8e57961cf67 with delete_type TABLET_DATA_DELETED (replaced by tablet 2312c1ee4a134caaa68980a224dd283e) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:31.811344 1456 catalog_manager.cc:6596] ReplaceTablet: tablet 6598311f59c6476ba339b8e57961cf67 deleted and replaced by tablet 2312c1ee4a134caaa68980a224dd283e
I20250110 02:51:31.811447 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 176) took 2018 ms. Trace:
I20250110 02:51:31.811757 2357 ts_tablet_manager.cc:1905] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.811527 1456 rpcz_store.cc:276] 0110 02:51:29.792889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.754337 (+1961448us) service_pool.cc:225] Handling call
0110 02:51:31.811421 (+ 57084us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.805929 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.806013 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.806018 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.806021 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:31.806112 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.806130 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.806134 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.806203 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:31.806205 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.806229 (+ 24us) write_op.cc:270] Start()
0110 02:51:31.806251 (+ 22us) write_op.cc:276] Timestamp: P: 1736477491806184 usec, L: 0
0110 02:51:31.806254 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.806402 (+ 148us) log.cc:844] Serialized 622 byte log entry
0110 02:51:31.808889 (+ 2487us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.808988 (+ 99us) write_op.cc:301] APPLY: starting
0110 02:51:31.809000 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.809036 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.809062 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.809170 (+ 108us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.809221 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.809228 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.809287 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.809339 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.809357 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:31.809364 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.809425 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.809429 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":429,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":546,"prepare.run_wall_time_us":546,"raft.queue_time_us":321,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":168,"replication_time_us":2610}]]}
I20250110 02:51:31.812040 2549 raft_consensus.cc:491] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.812160 1456 master_service.cc:946] ReplaceTablet: received request to replace tablet 55edfa05075b4edcb70ef29e5ee413c6 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:31.812142 2549 raft_consensus.cc:513] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.812495 1456 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 177) took 1919 ms. Trace:
I20250110 02:51:31.812569 1456 rpcz_store.cc:276] 0110 02:51:29.893003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.812098 (+1919095us) service_pool.cc:225] Handling call
0110 02:51:31.812452 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 02:51:31.812727 2549 leader_election.cc:290] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.812829 1456 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_ed1ab72a595140a98cef7ad004d2c981" } modify_external_catalogs: true
I20250110 02:51:31.812973 1456 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_ed1ab72a595140a98cef7ad004d2c981" } modify_external_catalogs: true
I20250110 02:51:31.813237 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.813781 709 leader_election.cc:343] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.813855 2378 raft_consensus.cc:491] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.813974 2378 raft_consensus.cc:513] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.814487 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6598311f59c6476ba339b8e57961cf67 with delete_type TABLET_DATA_DELETED (replaced by tablet 2312c1ee4a134caaa68980a224dd283e) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:31.814582 2378 leader_election.cc:290] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:31.815181 853 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
W20250110 02:51:31.815846 853 leader_election.cc:343] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:31.815949 853 leader_election.cc:304] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:31.816212 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 221) took 5500 ms. Trace:
I20250110 02:51:31.816282 2352 rpcz_store.cc:276] 0110 02:51:26.315811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317612 (+ 1801us) service_pool.cc:225] Handling call
0110 02:51:31.597143 (+5279531us) ts_tablet_manager.cc:1145] Deleting tablet 8f15078fc22f4b32a48c7190b36e2192
0110 02:51:31.597154 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.695800 (+ 98646us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.724880 (+ 29080us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.816198 (+ 91318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5279424}
I20250110 02:51:31.816577 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 223) took 5500 ms. Trace:
I20250110 02:51:31.816625 2352 rpcz_store.cc:276] 0110 02:51:26.315961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317730 (+ 1769us) service_pool.cc:225] Handling call
0110 02:51:31.816496 (+5498766us) ts_tablet_manager.cc:1145] Deleting tablet b4f1782c46a44ed091f4c533226c71f2
0110 02:51:31.816503 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.816569 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5498660}
I20250110 02:51:31.816789 2352 tablet_replica.cc:331] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.816931 2352 raft_consensus.cc:2238] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.817144 2378 raft_consensus.cc:2743] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.817206 2352 raft_consensus.cc:2267] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.817634 2352 ts_tablet_manager.cc:1905] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.818614 1188 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:31.820072 1188 tablet_bootstrap.cc:654] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:31.771718 996 consensus_peers.cc:487] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.777681 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.820705 996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 49020.1us
W20250110 02:51:31.820809 711 leader_election.cc:343] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.820892 711 leader_election.cc:304] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.821498 2549 raft_consensus.cc:2743] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:31.821764 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:31.822198 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 277) took 4382 ms. Trace:
I20250110 02:51:31.822275 2224 rpcz_store.cc:276] 0110 02:51:27.439910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.440555 (+ 645us) service_pool.cc:225] Handling call
0110 02:51:31.822099 (+4381544us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:31.822107 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.822168 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4381466}
W20250110 02:51:31.822633 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because the tablet was not found. No further retry: Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:31.822911 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 278) took 4373 ms. Trace:
I20250110 02:51:31.822983 2224 rpcz_store.cc:276] 0110 02:51:27.449507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.449554 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:31.822827 (+4373273us) ts_tablet_manager.cc:1145] Deleting tablet 8c3ce95413df40369ee599176f91b892
0110 02:51:31.822835 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.822882 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4373107}
W20250110 02:51:31.823318 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8c3ce95413df40369ee599176f91b892 because the tablet was not found. No further retry: Not found: Tablet not found: 8c3ce95413df40369ee599176f91b892
I20250110 02:51:31.823547 2224 tablet_replica.cc:331] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:31.823725 2224 raft_consensus.cc:2238] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.823874 2224 raft_consensus.cc:2267] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.824470 2224 ts_tablet_manager.cc:1905] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.825695 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 8f15078fc22f4b32a48c7190b36e2192 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]) successfully deleted
W20250110 02:51:31.825876 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:31.826063 2456 consensus_queue.cc:1035] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.751196 2738 consensus_queue.cc:1035] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.837541 2407 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "85f9597077934f1a9814d284ed06d13d" instance_seqno: 1736477467615051) as {username='slave'} at 127.0.104.2:59821; Asking this server to re-register.
I20250110 02:51:31.837854 1184 tablet_bootstrap.cc:492] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.837958 1184 ts_tablet_manager.cc:1397] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.000s
I20250110 02:51:31.838780 1184 raft_consensus.cc:357] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.839056 1184 raft_consensus.cc:383] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.839164 1184 raft_consensus.cc:738] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.839469 1184 consensus_queue.cc:260] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.811275 1181 tablet_bootstrap.cc:492] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.813784 1106 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:31.814183 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6598311f59c6476ba339b8e57961cf67 with delete_type TABLET_DATA_DELETED (replaced by tablet 2312c1ee4a134caaa68980a224dd283e) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:31.840046 1181 ts_tablet_manager.cc:1397] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.000s
I20250110 02:51:31.840096 1184 ts_tablet_manager.cc:1428] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:31.814354 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.840265 2352 ts_tablet_manager.cc:1918] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.840042 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
W20250110 02:51:31.840498 711 leader_election.cc:343] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.840587 711 leader_election.cc:304] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.840778 1181 raft_consensus.cc:357] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.840981 1181 raft_consensus.cc:383] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.841030 1184 tablet_bootstrap.cc:492] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:31.841046 1181 raft_consensus.cc:738] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.842588 2352 log.cc:1198] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:31.842530 1181 consensus_queue.cc:260] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:31.842896 2352 ts_tablet_manager.cc:1939] T 323ce9f6a3bb4e4a8d80ca0b047500a5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.843111 1184 tablet_bootstrap.cc:654] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.843344 1181 ts_tablet_manager.cc:1428] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:31.843164 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 226) took 5526 ms. Trace:
I20250110 02:51:31.843498 2352 rpcz_store.cc:276] 0110 02:51:26.316224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.317863 (+ 1639us) service_pool.cc:225] Handling call
0110 02:51:31.816760 (+5498897us) ts_tablet_manager.cc:1145] Deleting tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
0110 02:51:31.816766 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.828634 (+ 11868us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.840184 (+ 11550us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.843150 (+ 2966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5498776}
I20250110 02:51:31.843622 1181 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.843906 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]) successfully deleted
I20250110 02:51:31.844198 2352 tablet_replica.cc:331] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:31.844359 2352 raft_consensus.cc:2238] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.844625 2352 raft_consensus.cc:2267] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.846414 2549 raft_consensus.cc:2743] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.846748 965 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:31.854146 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 746 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:31.856655 1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 746 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:31.857722 2407 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" instance_seqno: 1736477468177459) as {username='slave'} at 127.0.104.3:47545; Asking this server to re-register.
I20250110 02:51:31.858237 2549 raft_consensus.cc:491] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.858446 2549 raft_consensus.cc:513] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.859205 2549 leader_election.cc:290] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.859869 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55edfa05075b4edcb70ef29e5ee413c6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.859941 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55edfa05075b4edcb70ef29e5ee413c6" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.860612 709 leader_election.cc:343] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:31.860829 711 leader_election.cc:343] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.860922 711 leader_election.cc:304] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.861119 2549 raft_consensus.cc:2743] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.846954 2352 ts_tablet_manager.cc:1905] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.867808 2549 raft_consensus.cc:491] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.857210 914 raft_consensus.cc:3054] T 069d19975e2a4bb0a23b4e9983f411d3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.867942 2549 raft_consensus.cc:513] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.868749 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.869592 709 leader_election.cc:343] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.869768 2549 leader_election.cc:290] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.870244 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:31.870677 711 leader_election.cc:343] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.870774 711 leader_election.cc:304] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:31.871156 2549 raft_consensus.cc:2743] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.871549 2906 consensus_queue.cc:1035] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.874944 1181 tablet_bootstrap.cc:654] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.861807 2295 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8bc934cc708045e9858ee8f8e64660cd; no voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558
I20250110 02:51:31.883728 2357 ts_tablet_manager.cc:1918] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.883858 3059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 02:51:31.886776 2378 raft_consensus.cc:491] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.886986 2378 raft_consensus.cc:513] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.887369 2357 log.cc:1198] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/d761ef0d1e5542b1979c29bb2d579489
I20250110 02:51:31.887691 2378 leader_election.cc:290] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.888087 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.889364 2357 ts_tablet_manager.cc:1939] T d761ef0d1e5542b1979c29bb2d579489 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.897495 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 188) took 4412 ms. Trace:
I20250110 02:51:31.897707 2357 rpcz_store.cc:276] 0110 02:51:27.485248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.485717 (+ 469us) service_pool.cc:225] Handling call
0110 02:51:31.786632 (+4300915us) ts_tablet_manager.cc:1145] Deleting tablet d761ef0d1e5542b1979c29bb2d579489
0110 02:51:31.786640 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.834839 (+ 48199us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.883616 (+ 48777us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.897474 (+ 13858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4300123}
I20250110 02:51:31.900187 3014 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.888281 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84adc1fd3f76497b98a0fcbc52a3120a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.899780 2357 tablet_replica.cc:331] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:31.902599 853 leader_election.cc:304] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:31.902818 2378 raft_consensus.cc:2743] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:31.905001 853 consensus_peers.cc:487] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.905937 1184 tablet_bootstrap.cc:492] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:31.893940 2224 ts_tablet_manager.cc:1918] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.899503 3012 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:31.910662 2224 log.cc:1198] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:31.911180 2224 ts_tablet_manager.cc:1939] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.911337 1184 ts_tablet_manager.cc:1397] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250110 02:51:31.911211 1108 heartbeater.cc:463] Registering TS with master...
I20250110 02:51:31.912104 1184 raft_consensus.cc:357] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.912292 1184 raft_consensus.cc:383] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.912351 1184 raft_consensus.cc:738] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.893054 2410 ts_manager.cc:194] Registered new tserver with Master: 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:31.916460 2357 raft_consensus.cc:2238] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.916777 2357 pending_rounds.cc:62] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Trying to abort 1 pending ops.
I20250110 02:51:31.916854 2357 pending_rounds.cc:69] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611803036151808 op_type: NO_OP noop_request { }
I20250110 02:51:31.916970 2357 raft_consensus.cc:2883] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:31.917090 2357 raft_consensus.cc:2267] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.917430 2410 ts_manager.cc:194] Registered new tserver with Master: 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.920513 2357 ts_tablet_manager.cc:1905] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:31.921983 996 consensus_peers.cc:487] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:31.923259 994 consensus_peers.cc:487] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:31.923468 2905 raft_consensus.cc:491] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.923612 2905 raft_consensus.cc:513] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.924079 2905 leader_election.cc:290] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.924495 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.924893 994 leader_election.cc:343] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.927508 2357 ts_tablet_manager.cc:1918] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.928158 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 280) took 4472 ms. Trace:
I20250110 02:51:31.928375 2224 rpcz_store.cc:276] 0110 02:51:27.455502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.455682 (+ 180us) service_pool.cc:225] Handling call
0110 02:51:31.823494 (+4367812us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:31.823502 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.866013 (+ 42511us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.893826 (+ 27813us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.928128 (+ 34302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4367653}
I20250110 02:51:31.928810 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 19b90cc15dbf47d4b4dbaf12d588abb0 (table default.Table_b735b26f52fc4d4d9a3f706c984d008e [id=5cff2f6d15f14bed8582bee898c063da]) successfully deleted
I20250110 02:51:31.928915 2224 tablet_replica.cc:331] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:31.929119 2224 raft_consensus.cc:2238] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:31.929282 2224 raft_consensus.cc:2267] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.929809 2224 ts_tablet_manager.cc:1905] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:31.932588 2905 consensus_queue.cc:1035] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:31.931820 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:31.939347 771 raft_consensus.cc:2463] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:31.939654 2357 log.cc:1198] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f9b379b357194d729108428cbaebd51a
I20250110 02:51:31.940049 2357 ts_tablet_manager.cc:1939] T f9b379b357194d729108428cbaebd51a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:31.941056 996 leader_election.cc:304] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.941339 2905 raft_consensus.cc:2798] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:31.941413 2905 raft_consensus.cc:491] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:31.941481 2905 raft_consensus.cc:3054] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.942251 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 175) took 4461 ms. Trace:
I20250110 02:51:31.942317 2357 rpcz_store.cc:276] 0110 02:51:27.480371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.480414 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:31.898710 (+4418296us) ts_tablet_manager.cc:1145] Deleting tablet f9b379b357194d729108428cbaebd51a
0110 02:51:31.898721 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.916693 (+ 17972us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x6160001c5a54. stack: 00007fbee81fbfd6 00007fbee81fbd90 00007fbee71db48d 000000000096d544 000000000096d50d 000000000096ceda 00007fbef3a9eed0 00007fbef4d93463 00007fbef7837eb7 00007fbef786a584 00007fbef784b690 00007fbef784b41d 0000000000a71ecc 00007fbee8242b64 00007fbee824618b 00007fbee8246008
0110 02:51:31.924947 (+ 8254us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.927444 (+ 2497us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.942235 (+ 14791us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34273536,"tablet-delete.queue_time_us":4412155}
I20250110 02:51:31.942832 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f9b379b357194d729108428cbaebd51a (table default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]) successfully deleted
I20250110 02:51:31.944060 2905 raft_consensus.cc:513] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.944919 2905 leader_election.cc:290] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.945168 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:31.945616 994 leader_election.cc:343] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.945837 893 tablet_service.cc:1467] Processing CreateTablet for tablet f779879503a24c1286a941cc2c498fc6 (DEFAULT_TABLE table=default.Table_e027fefdb6324ebabe8604204d2a6712 [id=b857dd90402542669793e4733dd36b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.946182 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f779879503a24c1286a941cc2c498fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.946262 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f779879503a24c1286a941cc2c498fc6 (DEFAULT_TABLE table=default.Table_e027fefdb6324ebabe8604204d2a6712 [id=b857dd90402542669793e4733dd36b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.946604 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f779879503a24c1286a941cc2c498fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.950049 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b538100977456794c9519bbdffa7e7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:31.950219 772 raft_consensus.cc:3054] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:31.952530 772 raft_consensus.cc:2463] T 54b538100977456794c9519bbdffa7e7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:31.956446 2905 raft_consensus.cc:491] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.956616 2905 raft_consensus.cc:513] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.957201 2352 ts_tablet_manager.cc:1918] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:31.957217 2905 leader_election.cc:290] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.957369 2352 log.cc:1198] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d424b4da66184bd19eccefad80621516
I20250110 02:51:31.957742 2352 ts_tablet_manager.cc:1939] T d424b4da66184bd19eccefad80621516 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:31.957718 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7ec8c7891b4a2c9fcb3cc2eeedd29a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:31.958141 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 232) took 5639 ms. Trace:
W20250110 02:51:31.958254 994 leader_election.cc:343] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.959645 996 leader_election.cc:304] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:31.960769 756 tablet_service.cc:1467] Processing CreateTablet for tablet f779879503a24c1286a941cc2c498fc6 (DEFAULT_TABLE table=default.Table_e027fefdb6324ebabe8604204d2a6712 [id=b857dd90402542669793e4733dd36b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.961133 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f779879503a24c1286a941cc2c498fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.960109 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f92f583308094c1dac8595ab7d4a070b (DEFAULT_TABLE table=default.Table_cb3b2b01aabd4f929eec83b441138ffc [id=a22e2117fbbb4b35bfbb2f11b279fc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.963490 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92f583308094c1dac8595ab7d4a070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.968808 896 tablet_service.cc:1467] Processing CreateTablet for tablet f92f583308094c1dac8595ab7d4a070b (DEFAULT_TABLE table=default.Table_cb3b2b01aabd4f929eec83b441138ffc [id=a22e2117fbbb4b35bfbb2f11b279fc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.969009 2289 raft_consensus.cc:2798] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:31.969156 2289 raft_consensus.cc:695] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:31.969384 2289 consensus_queue.cc:237] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.969794 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92f583308094c1dac8595ab7d4a070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.974409 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 189) took 4489 ms. Trace:
I20250110 02:51:31.974510 2357 rpcz_store.cc:276] 0110 02:51:27.485355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.485981 (+ 626us) service_pool.cc:225] Handling call
0110 02:51:31.974289 (+4488308us) ts_tablet_manager.cc:1145] Deleting tablet 8f15078fc22f4b32a48c7190b36e2192
0110 02:51:31.974298 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.974374 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4487633}
W20250110 02:51:31.974987 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because the tablet was not found. No further retry: Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:31.975368 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 190) took 4488 ms. Trace:
I20250110 02:51:31.975466 2357 rpcz_store.cc:276] 0110 02:51:27.487125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.487471 (+ 346us) service_pool.cc:225] Handling call
0110 02:51:31.975292 (+4487821us) ts_tablet_manager.cc:1145] Deleting tablet b375a932156d4abb908dbf327689b0f9
0110 02:51:31.975300 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.975353 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4486603}
I20250110 02:51:31.975603 2224 ts_tablet_manager.cc:1918] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:31.975886 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b375a932156d4abb908dbf327689b0f9 because the tablet was not found. No further retry: Not found: Tablet not found: b375a932156d4abb908dbf327689b0f9
I20250110 02:51:31.976318 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 192) took 4488 ms. Trace:
I20250110 02:51:31.976433 2357 rpcz_store.cc:276] 0110 02:51:27.487347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.487540 (+ 193us) service_pool.cc:225] Handling call
0110 02:51:31.976205 (+4488665us) ts_tablet_manager.cc:1145] Deleting tablet 76833ce515cf4ee9bc8fff0f626af220
0110 02:51:31.976222 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.976304 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4487398}
W20250110 02:51:31.976783 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 76833ce515cf4ee9bc8fff0f626af220 because the tablet was not found. No further retry: Not found: Tablet not found: 76833ce515cf4ee9bc8fff0f626af220
I20250110 02:51:31.976502 1184 consensus_queue.cc:260] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.976914 2549 raft_consensus.cc:491] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:31.977068 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 191) took 4489 ms. Trace:
I20250110 02:51:31.977039 2549 raft_consensus.cc:513] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.977128 2357 rpcz_store.cc:276] 0110 02:51:27.487227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.487445 (+ 218us) service_pool.cc:225] Handling call
0110 02:51:31.976996 (+4489551us) ts_tablet_manager.cc:1145] Deleting tablet 28340f0f9a2043a199c74f13755eb702
0110 02:51:31.977004 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.977053 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4486434}
W20250110 02:51:31.977455 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 28340f0f9a2043a199c74f13755eb702 because the tablet was not found. No further retry: Not found: Tablet not found: 28340f0f9a2043a199c74f13755eb702
I20250110 02:51:31.977963 2549 leader_election.cc:290] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:31.978622 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c0808330e848249e1d7eb71ab0d362" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:31.979416 709 leader_election.cc:343] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.980381 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 194) took 4486 ms. Trace:
I20250110 02:51:31.980576 1184 ts_tablet_manager.cc:1428] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20250110 02:51:31.980126 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c0808330e848249e1d7eb71ab0d362" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:31.981156 2224 log.cc:1198] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/452223af5f6a4836a0f582d394a9453e
I20250110 02:51:31.981527 2224 ts_tablet_manager.cc:1939] T 452223af5f6a4836a0f582d394a9453e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:31.982761 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 283) took 4527 ms. Trace:
I20250110 02:51:31.983686 756 tablet_service.cc:1467] Processing CreateTablet for tablet f92f583308094c1dac8595ab7d4a070b (DEFAULT_TABLE table=default.Table_cb3b2b01aabd4f929eec83b441138ffc [id=a22e2117fbbb4b35bfbb2f11b279fc45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:31.984052 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92f583308094c1dac8595ab7d4a070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:31.896162 2407 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "98336b684b41407da8a1949e30cd82e0" instance_seqno: 1736477467131857) as {username='slave'} at 127.0.104.1:35143; Asking this server to re-register.
W20250110 02:51:31.985318 996 leader_election.cc:343] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
I20250110 02:51:31.985527 2224 rpcz_store.cc:276] 0110 02:51:27.455758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.455808 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:31.928845 (+4473037us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:31.928854 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.971980 (+ 43126us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.975508 (+ 3528us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.982732 (+ 7224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4472901}
I20250110 02:51:31.985657 996 leader_election.cc:304] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:31.985915 2456 raft_consensus.cc:2743] T 1b7ec8c7891b4a2c9fcb3cc2eeedd29a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:31.985924 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 452223af5f6a4836a0f582d394a9453e (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]) successfully deleted
I20250110 02:51:31.986238 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 281) took 4530 ms. Trace:
I20250110 02:51:31.986328 2224 rpcz_store.cc:276] 0110 02:51:27.455574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.455876 (+ 302us) service_pool.cc:225] Handling call
0110 02:51:31.986137 (+4530261us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:31.986145 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.986201 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4530155}
I20250110 02:51:31.986675 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 284) took 4530 ms. Trace:
W20250110 02:51:31.986747 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because the tablet was not found. No further retry: Not found: Tablet not found: 8f4778347f054b86af697dbe3b287263
I20250110 02:51:31.986773 2224 rpcz_store.cc:276] 0110 02:51:27.455840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.455969 (+ 129us) service_pool.cc:225] Handling call
0110 02:51:31.986592 (+4530623us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:31.986599 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.986646 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4530518}
I20250110 02:51:31.987066 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 285) took 4531 ms. Trace:
W20250110 02:51:31.987123 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5e183122d13a4c38988bf2fe6f7a3694 because the tablet was not found. No further retry: Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:31.987147 2224 rpcz_store.cc:276] 0110 02:51:27.455921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456001 (+ 80us) service_pool.cc:225] Handling call
0110 02:51:31.986961 (+4530960us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:31.986969 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.987035 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4530872}
I20250110 02:51:31.987423 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 286) took 4531 ms. Trace:
I20250110 02:51:31.987514 824 heartbeater.cc:463] Registering TS with master...
W20250110 02:51:31.987493 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:31.987509 2224 rpcz_store.cc:276] 0110 02:51:27.455990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456111 (+ 121us) service_pool.cc:225] Handling call
0110 02:51:31.987326 (+4531215us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:31.987334 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.987395 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531125}
I20250110 02:51:31.987802 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 282) took 4532 ms. Trace:
W20250110 02:51:31.987923 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
I20250110 02:51:31.987885 2224 rpcz_store.cc:276] 0110 02:51:27.455658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456210 (+ 552us) service_pool.cc:225] Handling call
0110 02:51:31.987687 (+4531477us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:31.987696 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.987766 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531389}
I20250110 02:51:31.988200 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 289) took 4531 ms. Trace:
W20250110 02:51:31.988306 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
I20250110 02:51:31.988274 2224 rpcz_store.cc:276] 0110 02:51:27.456223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456312 (+ 89us) service_pool.cc:225] Handling call
0110 02:51:31.988081 (+4531769us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:31.988089 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.988167 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531644}
I20250110 02:51:31.988530 2224 tablet_replica.cc:331] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:31.988698 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
I20250110 02:51:31.988740 2224 raft_consensus.cc:2238] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:31.989041 2224 raft_consensus.cc:2267] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:31.989557 1181 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:31.990487 1181 ts_tablet_manager.cc:1397] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.000s
I20250110 02:51:31.991659 1181 raft_consensus.cc:357] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.992487 2407 ts_manager.cc:194] Registered new tserver with Master: 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:31.980481 2357 rpcz_store.cc:276] 0110 02:51:27.494386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.494507 (+ 121us) service_pool.cc:225] Handling call
0110 02:51:31.980309 (+4485802us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:31.980316 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.980366 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4485588}
W20250110 02:51:31.981031 711 leader_election.cc:343] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:31.992682 711 leader_election.cc:304] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:31.992878 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d9018955f3884824beb2e7336987fc1c because the tablet was not found. No further retry: Not found: Tablet not found: d9018955f3884824beb2e7336987fc1c
I20250110 02:51:31.993170 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 193) took 4499 ms. Trace:
I20250110 02:51:31.993305 2357 rpcz_store.cc:276] 0110 02:51:27.494259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.494585 (+ 326us) service_pool.cc:225] Handling call
0110 02:51:31.993074 (+4498489us) ts_tablet_manager.cc:1145] Deleting tablet d20177229f7b4f11875e0007afb1dd00
0110 02:51:31.993081 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.993144 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4498291}
I20250110 02:51:31.993484 1181 raft_consensus.cc:383] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:31.993566 1181 raft_consensus.cc:738] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:31.993820 2549 raft_consensus.cc:2743] T 78c0808330e848249e1d7eb71ab0d362 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:31.994606 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d20177229f7b4f11875e0007afb1dd00 because the tablet was not found. No further retry: Not found: Tablet not found: d20177229f7b4f11875e0007afb1dd00
I20250110 02:51:31.994890 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 195) took 4468 ms. Trace:
I20250110 02:51:31.994968 2357 rpcz_store.cc:276] 0110 02:51:27.526406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.526453 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:31.994796 (+4468343us) ts_tablet_manager.cc:1145] Deleting tablet d6fd370138ac4039be55642f1b4c5607
0110 02:51:31.994804 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.994875 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4468098}
W20250110 02:51:31.995313 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d6fd370138ac4039be55642f1b4c5607 because the tablet was not found. No further retry: Not found: Tablet not found: d6fd370138ac4039be55642f1b4c5607
I20250110 02:51:31.993762 1181 consensus_queue.cc:260] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:31.995934 1181 ts_tablet_manager.cc:1428] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250110 02:51:31.998204 1181 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:31.998495 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 196) took 4470 ms. Trace:
I20250110 02:51:31.998605 2357 rpcz_store.cc:276] 0110 02:51:27.527583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.527790 (+ 207us) service_pool.cc:225] Handling call
0110 02:51:31.998411 (+4470621us) ts_tablet_manager.cc:1145] Deleting tablet 19e4af23720048fe98b3231ba402d45c
0110 02:51:31.998419 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.998483 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4470409}
W20250110 02:51:31.999791 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because the tablet was not found. No further retry: Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:32.000088 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 197) took 4472 ms. Trace:
I20250110 02:51:32.000162 2357 rpcz_store.cc:276] 0110 02:51:27.527752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.527822 (+ 70us) service_pool.cc:225] Handling call
0110 02:51:32.000019 (+4472197us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:32.000027 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.000076 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4472001}
W20250110 02:51:32.000473 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because the tablet was not found. No further retry: Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:32.000727 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 198) took 4472 ms. Trace:
I20250110 02:51:32.000802 2357 rpcz_store.cc:276] 0110 02:51:27.528191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.528432 (+ 241us) service_pool.cc:225] Handling call
0110 02:51:32.000660 (+4472228us) ts_tablet_manager.cc:1145] Deleting tablet 4d970e2fe72b4727968ca605b3e772cc
0110 02:51:32.000668 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.000714 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4472016}
W20250110 02:51:32.001127 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because the tablet was not found. No further retry: Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:32.001609 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 199) took 4473 ms. Trace:
I20250110 02:51:32.001679 2357 rpcz_store.cc:276] 0110 02:51:27.528423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.528464 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.001532 (+4473068us) ts_tablet_manager.cc:1145] Deleting tablet 53ae5a6572bd4e7e90e92d4b1cf8d820
0110 02:51:32.001540 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.001596 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4472860}
W20250110 02:51:32.002010 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 53ae5a6572bd4e7e90e92d4b1cf8d820 because the tablet was not found. No further retry: Not found: Tablet not found: 53ae5a6572bd4e7e90e92d4b1cf8d820
I20250110 02:51:32.002295 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 200) took 4473 ms. Trace:
I20250110 02:51:32.002367 2357 rpcz_store.cc:276] 0110 02:51:27.528747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.528783 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.002209 (+4473426us) ts_tablet_manager.cc:1145] Deleting tablet 240b64d551a548e884ee2a7238c0363a
0110 02:51:32.002215 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.002280 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4473215}
W20250110 02:51:32.002693 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 240b64d551a548e884ee2a7238c0363a because the tablet was not found. No further retry: Not found: Tablet not found: 240b64d551a548e884ee2a7238c0363a
I20250110 02:51:32.002933 2357 tablet_replica.cc:331] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.003116 2357 raft_consensus.cc:2238] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.003268 2357 raft_consensus.cc:2267] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.003602 1181 tablet_bootstrap.cc:654] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.003774 2357 ts_tablet_manager.cc:1905] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.004957 896 tablet_service.cc:1467] Processing CreateTablet for tablet b684f75194374bc6ad93c59487c185e7 (DEFAULT_TABLE table=default.Table_7b4f8c49307a462c81559f8b98001b73 [id=be746a5c0435454fae03052ab1e7c139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.005358 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b684f75194374bc6ad93c59487c185e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.008927 2357 ts_tablet_manager.cc:1918] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:31.958256 2352 rpcz_store.cc:276] 0110 02:51:26.318196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.318334 (+ 138us) service_pool.cc:225] Handling call
0110 02:51:31.844156 (+5525822us) ts_tablet_manager.cc:1145] Deleting tablet d424b4da66184bd19eccefad80621516
0110 02:51:31.844165 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:31.898119 (+ 53954us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.957110 (+ 58991us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:31.958128 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5525483}
I20250110 02:51:32.011379 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d424b4da66184bd19eccefad80621516 (table default.Table_051f5b2057e84287934aac189c8ea3da [id=b47d2feaea9743c7aa78f87abe4231c6]) successfully deleted
I20250110 02:51:32.011691 2352 tablet_replica.cc:331] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.011921 2352 raft_consensus.cc:2238] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:32.012246 2352 raft_consensus.cc:2267] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.012754 2352 ts_tablet_manager.cc:1905] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.013944 2224 ts_tablet_manager.cc:1905] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.014926 1026 tablet_service.cc:1467] Processing CreateTablet for tablet b684f75194374bc6ad93c59487c185e7 (DEFAULT_TABLE table=default.Table_7b4f8c49307a462c81559f8b98001b73 [id=be746a5c0435454fae03052ab1e7c139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.015339 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b684f75194374bc6ad93c59487c185e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.023407 893 tablet_service.cc:1467] Processing CreateTablet for tablet 13326b995b524fc3b76c6154ac5f2c21 (DEFAULT_TABLE table=default.Table_88d75763367c40a2afacc85de418e9a4 [id=47cce9029d9942d1bb7c5f576e68e90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.023805 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13326b995b524fc3b76c6154ac5f2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.024405 1036 tablet_service.cc:1467] Processing CreateTablet for tablet e5cc4e2e4abf4a5faf55632a2b59381e (DEFAULT_TABLE table=default.Table_80f6fdd02e5349578c351409e97f722a [id=cf52652c28e2419daeb70d39b570d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.024757 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cc4e2e4abf4a5faf55632a2b59381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.025545 2691 consensus_queue.cc:1035] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.026655 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 13326b995b524fc3b76c6154ac5f2c21 (DEFAULT_TABLE table=default.Table_88d75763367c40a2afacc85de418e9a4 [id=47cce9029d9942d1bb7c5f576e68e90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.026979 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13326b995b524fc3b76c6154ac5f2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.031406 1181 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.031937 1181 ts_tablet_manager.cc:1397] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20250110 02:51:32.034984 2224 ts_tablet_manager.cc:1918] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.022590 747 tablet_service.cc:1467] Processing CreateTablet for tablet e5cc4e2e4abf4a5faf55632a2b59381e (DEFAULT_TABLE table=default.Table_80f6fdd02e5349578c351409e97f722a [id=cf52652c28e2419daeb70d39b570d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.035174 2357 log.cc:1198] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/55edfa05075b4edcb70ef29e5ee413c6
I20250110 02:51:32.035362 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cc4e2e4abf4a5faf55632a2b59381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.035524 2357 ts_tablet_manager.cc:1939] T 55edfa05075b4edcb70ef29e5ee413c6 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.036237 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 201) took 4506 ms. Trace:
I20250110 02:51:32.036625 2224 log.cc:1198] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/d761ef0d1e5542b1979c29bb2d579489
I20250110 02:51:32.037097 2224 ts_tablet_manager.cc:1939] T d761ef0d1e5542b1979c29bb2d579489 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.038448 2357 rpcz_store.cc:276] 0110 02:51:27.529607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.529647 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:32.002890 (+4473243us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:32.002898 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.006401 (+ 3503us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.008856 (+ 2455us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.036220 (+ 27364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4473050}
I20250110 02:51:32.036975 1181 raft_consensus.cc:357] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.038816 1181 raft_consensus.cc:383] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.038996 1181 raft_consensus.cc:738] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.039407 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 288) took 4583 ms. Trace:
I20250110 02:51:32.039789 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 55edfa05075b4edcb70ef29e5ee413c6 (table default.Table_7431ea53e3704aebb4d79e0a0c4045f5 [id=70ad733a8a4943a39c52ee59ad2253cb]) successfully deleted
I20250110 02:51:32.040292 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 202) took 4510 ms. Trace:
I20250110 02:51:32.040383 2357 rpcz_store.cc:276] 0110 02:51:27.529994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.530030 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.040195 (+4510165us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:32.040204 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.040277 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4509987}
W20250110 02:51:32.040750 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because the tablet was not found. No further retry: Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
I20250110 02:51:32.041082 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 203) took 4510 ms. Trace:
I20250110 02:51:32.042114 2357 rpcz_store.cc:276] 0110 02:51:27.530551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.530587 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.040994 (+4510407us) ts_tablet_manager.cc:1145] Deleting tablet 3fb805b3e537466ab7fe0185421ad5aa
0110 02:51:32.041002 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.041070 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4510227}
I20250110 02:51:32.039362 1181 consensus_queue.cc:260] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.042843 1181 ts_tablet_manager.cc:1428] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250110 02:51:32.043183 1181 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.044965 1181 tablet_bootstrap.cc:654] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.051568 2357 tablet_replica.cc:331] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.051718 914 raft_consensus.cc:3054] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.051790 2357 raft_consensus.cc:2238] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.051923 2357 raft_consensus.cc:2267] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:32.052723 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:32.042320 2224 rpcz_store.cc:276] 0110 02:51:27.456142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456454 (+ 312us) service_pool.cc:225] Handling call
0110 02:51:31.988447 (+4531993us) ts_tablet_manager.cc:1145] Deleting tablet d761ef0d1e5542b1979c29bb2d579489
0110 02:51:31.988456 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.016654 (+ 28198us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.034889 (+ 18235us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.039363 (+ 4474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531902}
I20250110 02:51:32.057906 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 291) took 4601 ms. Trace:
I20250110 02:51:32.058032 2224 rpcz_store.cc:276] 0110 02:51:27.456417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456452 (+ 35us) service_pool.cc:225] Handling call
0110 02:51:32.057568 (+4601116us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:32.057579 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.057870 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4600980}
W20250110 02:51:32.058488 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 0127de45264f4660b843734a530eb310 because the tablet was not found. No further retry: Not found: Tablet not found: 0127de45264f4660b843734a530eb310
I20250110 02:51:32.058921 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 287) took 4602 ms. Trace:
I20250110 02:51:32.059046 2224 rpcz_store.cc:276] 0110 02:51:27.456058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456103 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:32.058802 (+4602699us) ts_tablet_manager.cc:1145] Deleting tablet a7397e41f7374e9fb1621dbcf30af703
0110 02:51:32.058812 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.058874 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4602184}
I20250110 02:51:32.023852 756 tablet_service.cc:1467] Processing CreateTablet for tablet b684f75194374bc6ad93c59487c185e7 (DEFAULT_TABLE table=default.Table_7b4f8c49307a462c81559f8b98001b73 [id=be746a5c0435454fae03052ab1e7c139]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:32.059530 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a7397e41f7374e9fb1621dbcf30af703 because the tablet was not found. No further retry: Not found: Tablet not found: a7397e41f7374e9fb1621dbcf30af703
I20250110 02:51:32.059980 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b684f75194374bc6ad93c59487c185e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.060214 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 293) took 4603 ms. Trace:
I20250110 02:51:32.060329 2224 rpcz_store.cc:276] 0110 02:51:27.456592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456645 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:32.060108 (+4603463us) ts_tablet_manager.cc:1145] Deleting tablet 60e534bf66f044b88d8cda0f3a1228f4
0110 02:51:32.060117 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.060177 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4603342}
I20250110 02:51:32.060421 2357 ts_tablet_manager.cc:1905] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:32.066059 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 60e534bf66f044b88d8cda0f3a1228f4 because the tablet was not found. No further retry: Not found: Tablet not found: 60e534bf66f044b88d8cda0f3a1228f4
I20250110 02:51:32.066284 2357 ts_tablet_manager.cc:1918] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.066438 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 296) took 4609 ms. Trace:
I20250110 02:51:32.066540 2224 rpcz_store.cc:276] 0110 02:51:27.456874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.456924 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:32.066305 (+4609381us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:32.066314 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.066408 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4609263}
W20250110 02:51:32.070879 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet dad0d561a8ef498ba13bc813d4cd82fa because the tablet was not found. No further retry: Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:32.071472 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 297) took 4614 ms. Trace:
I20250110 02:51:32.071591 2224 rpcz_store.cc:276] 0110 02:51:27.456973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457066 (+ 93us) service_pool.cc:225] Handling call
0110 02:51:32.071386 (+4614320us) ts_tablet_manager.cc:1145] Deleting tablet d10ace37b96844328ccdb821dad3c5ea
0110 02:51:32.071395 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.071444 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614175}
W20250110 02:51:32.071956 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d10ace37b96844328ccdb821dad3c5ea because the tablet was not found. No further retry: Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:32.072256 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 290) took 4615 ms. Trace:
I20250110 02:51:32.072365 2224 rpcz_store.cc:276] 0110 02:51:27.456315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457170 (+ 855us) service_pool.cc:225] Handling call
0110 02:51:32.072188 (+4615018us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:32.072197 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.072243 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614869}
W20250110 02:51:32.072796 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2ef72e05f8ed4d46b759c7a04497653b because the tablet was not found. No further retry: Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b
I20250110 02:51:32.073285 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 299) took 4616 ms. Trace:
I20250110 02:51:32.073405 2224 rpcz_store.cc:276] 0110 02:51:27.457152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457235 (+ 83us) service_pool.cc:225] Handling call
0110 02:51:32.073180 (+4615945us) ts_tablet_manager.cc:1145] Deleting tablet 2e25f9643ac0476a90c93fb9aeb921a3
0110 02:51:32.073187 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.073236 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4615835}
W20250110 02:51:32.073750 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because the tablet was not found. No further retry: Not found: Tablet not found: 2e25f9643ac0476a90c93fb9aeb921a3
I20250110 02:51:32.074081 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 292) took 4617 ms. Trace:
I20250110 02:51:32.074201 2224 rpcz_store.cc:276] 0110 02:51:27.456509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457343 (+ 834us) service_pool.cc:225] Handling call
0110 02:51:32.073967 (+4616624us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:32.073975 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.074045 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4616502}
W20250110 02:51:32.074613 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:32.074898 2224 tablet_replica.cc:331] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.075119 2224 raft_consensus.cc:2238] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:32.075525 2224 raft_consensus.cc:2267] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.076056 2224 ts_tablet_manager.cc:1905] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.078212 896 tablet_service.cc:1467] Processing CreateTablet for tablet e5cc4e2e4abf4a5faf55632a2b59381e (DEFAULT_TABLE table=default.Table_80f6fdd02e5349578c351409e97f722a [id=cf52652c28e2419daeb70d39b570d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.080374 2980 mvcc.cc:204] Tried to move back new op lower bound from 7112611778655227904 to 7112611752058830848. Current Snapshot: MvccSnapshot[applied={T|T < 7112611778655227904 or (T in {7112611778655227904})}]
I20250110 02:51:32.081959 747 tablet_service.cc:1467] Processing CreateTablet for tablet 13326b995b524fc3b76c6154ac5f2c21 (DEFAULT_TABLE table=default.Table_88d75763367c40a2afacc85de418e9a4 [id=47cce9029d9942d1bb7c5f576e68e90a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.082325 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13326b995b524fc3b76c6154ac5f2c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.085866 2549 raft_consensus.cc:491] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.086014 2549 raft_consensus.cc:513] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.086687 2549 leader_election.cc:290] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:32.089565 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ddab9e847445999ec97eac4352d9c2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:32.093359 711 leader_election.cc:343] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.098760 2224 ts_tablet_manager.cc:1918] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.104139 2224 log.cc:1198] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c5601ca24b8f428e8a51cbeec3664901
I20250110 02:51:32.104547 2224 ts_tablet_manager.cc:1939] T c5601ca24b8f428e8a51cbeec3664901 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.106000 756 tablet_service.cc:1467] Processing CreateTablet for tablet 81138947fcab4db69bf1492de1f8560f (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.106022 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 81138947fcab4db69bf1492de1f8560f (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.106359 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81138947fcab4db69bf1492de1f8560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.106398 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81138947fcab4db69bf1492de1f8560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.106678 1184 tablet_bootstrap.cc:492] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.108664 1184 tablet_bootstrap.cc:654] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.078502 908 raft_consensus.cc:3054] T 8c7fbdab52014dddbf63cfef69b09b7b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.113966 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 300) took 4656 ms. Trace:
I20250110 02:51:32.114058 2224 rpcz_store.cc:276] 0110 02:51:27.457230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457367 (+ 137us) service_pool.cc:225] Handling call
0110 02:51:32.074837 (+4617470us) ts_tablet_manager.cc:1145] Deleting tablet c5601ca24b8f428e8a51cbeec3664901
0110 02:51:32.074845 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.095948 (+ 21103us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.098631 (+ 2683us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.113928 (+ 15297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4617327}
I20250110 02:51:32.114491 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c5601ca24b8f428e8a51cbeec3664901 (table default.Table_3375784ee4514b079aaf4ef21656d9ec [id=ce7dabed19974fa18c469d3fc981f4fc]) successfully deleted
I20250110 02:51:32.081182 1188 tablet_bootstrap.cc:492] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.087345 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ddab9e847445999ec97eac4352d9c2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.105259 893 tablet_service.cc:1467] Processing CreateTablet for tablet 81138947fcab4db69bf1492de1f8560f (DEFAULT_TABLE table=default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 02:51:32.115942 709 leader_election.cc:343] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.116050 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81138947fcab4db69bf1492de1f8560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.116089 709 leader_election.cc:304] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:32.116568 2549 raft_consensus.cc:2743] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.119645 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 298) took 4662 ms. Trace:
I20250110 02:51:32.119784 2224 rpcz_store.cc:276] 0110 02:51:27.457054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457536 (+ 482us) service_pool.cc:225] Handling call
0110 02:51:32.119507 (+4661971us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:32.119516 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.119604 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4661854}
W20250110 02:51:32.120229 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:32.120553 2224 tablet_replica.cc:331] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.120752 2224 raft_consensus.cc:2238] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.121798 2357 log.cc:1198] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/82bd4307e5fe4f61b9308005cf3446e1
I20250110 02:51:32.122486 2224 raft_consensus.cc:2267] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.122822 2357 ts_tablet_manager.cc:1939] T 82bd4307e5fe4f61b9308005cf3446e1 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.123253 2224 ts_tablet_manager.cc:1905] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.124480 1188 ts_tablet_manager.cc:1397] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.306s user 0.015s sys 0.000s
I20250110 02:51:32.125149 1188 raft_consensus.cc:357] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.125313 1188 raft_consensus.cc:383] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.125411 1188 raft_consensus.cc:738] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.125591 1188 consensus_queue.cc:260] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.126091 1188 ts_tablet_manager.cc:1428] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.126332 1188 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.127532 1188 tablet_bootstrap.cc:654] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.130600 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cc4e2e4abf4a5faf55632a2b59381e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.142952 893 tablet_service.cc:1467] Processing CreateTablet for tablet 20b952da88fc4590b71a8c36ff0e484f (DEFAULT_TABLE table=default.Table_001aba654de747868b42e5c672869b6e [id=a0d928a2c93640959f83ff54a12c8719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.143385 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b952da88fc4590b71a8c36ff0e484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.145602 2224 ts_tablet_manager.cc:1918] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.146731 2224 log.cc:1198] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a7d4f03a8c66446ea7c6d178cc7ef80a
I20250110 02:51:32.147063 2224 ts_tablet_manager.cc:1939] T a7d4f03a8c66446ea7c6d178cc7ef80a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.149314 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 294) took 4692 ms. Trace:
I20250110 02:51:32.149451 2224 rpcz_store.cc:276] 0110 02:51:27.456690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457669 (+ 979us) service_pool.cc:225] Handling call
0110 02:51:32.120490 (+4662821us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:32.120499 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.129380 (+ 8881us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.145499 (+ 16119us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.149267 (+ 3768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4662695}
I20250110 02:51:32.149883 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a7d4f03a8c66446ea7c6d178cc7ef80a (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:32.151027 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd9b07cb5604fed8b3eb1b576353940 (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.151367 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd9b07cb5604fed8b3eb1b576353940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.151487 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 301) took 4694 ms. Trace:
I20250110 02:51:32.151585 2224 rpcz_store.cc:276] 0110 02:51:27.457312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.457684 (+ 372us) service_pool.cc:225] Handling call
0110 02:51:32.151357 (+4693673us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:32.151366 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.151452 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4693539}
W20250110 02:51:32.153113 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f66603a0ee8c4082807c5a6aabe7e339 because the tablet was not found. No further retry: Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:32.153751 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 302) took 4695 ms. Trace:
I20250110 02:51:32.154433 2224 rpcz_store.cc:276] 0110 02:51:27.457925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.458087 (+ 162us) service_pool.cc:225] Handling call
0110 02:51:32.153656 (+4695569us) ts_tablet_manager.cc:1145] Deleting tablet da8f247c862e4cb68452e230857ee6b0
0110 02:51:32.153664 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.153713 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4695427}
W20250110 02:51:32.154788 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet da8f247c862e4cb68452e230857ee6b0 because the tablet was not found. No further retry: Not found: Tablet not found: da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:32.155144 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 303) took 4696 ms. Trace:
I20250110 02:51:32.162617 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd9b07cb5604fed8b3eb1b576353940 (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.163024 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd9b07cb5604fed8b3eb1b576353940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.171842 2224 rpcz_store.cc:276] 0110 02:51:27.458344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.458496 (+ 152us) service_pool.cc:225] Handling call
0110 02:51:32.155000 (+4696504us) ts_tablet_manager.cc:1145] Deleting tablet 3fb805b3e537466ab7fe0185421ad5aa
0110 02:51:32.155008 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.155091 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4696360}
W20250110 02:51:32.172973 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
I20250110 02:51:32.157099 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 20b952da88fc4590b71a8c36ff0e484f (DEFAULT_TABLE table=default.Table_001aba654de747868b42e5c672869b6e [id=a0d928a2c93640959f83ff54a12c8719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.157995 1181 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.173375 1181 ts_tablet_manager.cc:1397] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.130s user 0.017s sys 0.019s
I20250110 02:51:32.173575 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 304) took 4714 ms. Trace:
I20250110 02:51:32.173679 2224 rpcz_store.cc:276] 0110 02:51:27.458797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.458995 (+ 198us) service_pool.cc:225] Handling call
0110 02:51:32.173341 (+4714346us) ts_tablet_manager.cc:1145] Deleting tablet 09f01daf4c67462ba75e1d0cf56cef12
0110 02:51:32.173442 (+ 101us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.173518 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4714195}
I20250110 02:51:32.150707 756 tablet_service.cc:1467] Processing CreateTablet for tablet 20b952da88fc4590b71a8c36ff0e484f (DEFAULT_TABLE table=default.Table_001aba654de747868b42e5c672869b6e [id=a0d928a2c93640959f83ff54a12c8719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.173576 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b952da88fc4590b71a8c36ff0e484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.174134 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b952da88fc4590b71a8c36ff0e484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.174172 1181 raft_consensus.cc:357] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.174345 1181 raft_consensus.cc:383] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.174450 1181 raft_consensus.cc:738] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.174659 1181 consensus_queue.cc:260] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.174754 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 204) took 4644 ms. Trace:
I20250110 02:51:32.174815 2357 rpcz_store.cc:276] 0110 02:51:27.530658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.530777 (+ 119us) service_pool.cc:225] Handling call
0110 02:51:32.051524 (+4520747us) ts_tablet_manager.cc:1145] Deleting tablet 82bd4307e5fe4f61b9308005cf3446e1
0110 02:51:32.051533 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.063802 (+ 12269us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.066213 (+ 2411us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.174741 (+108528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4520561}
I20250110 02:51:32.175109 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 205) took 4644 ms. Trace:
I20250110 02:51:32.175146 1181 ts_tablet_manager.cc:1428] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.176052 1181 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.177724 1181 tablet_bootstrap.cc:654] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.178555 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 82bd4307e5fe4f61b9308005cf3446e1 (table default.Table_cba45bffb2a3419c83fafeea7087a9f0 [id=86adcdf648c64d6d8f408103f844a1b2]) successfully deleted
W20250110 02:51:32.178879 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 09f01daf4c67462ba75e1d0cf56cef12 because the tablet was not found. No further retry: Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:32.171634 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd9b07cb5604fed8b3eb1b576353940 (DEFAULT_TABLE table=default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.181231 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd9b07cb5604fed8b3eb1b576353940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.181752 1188 tablet_bootstrap.cc:492] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.181941 1188 ts_tablet_manager.cc:1397] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.002s
I20250110 02:51:32.182559 1188 raft_consensus.cc:357] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.182790 1188 raft_consensus.cc:383] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.182895 1188 raft_consensus.cc:738] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.183115 1188 consensus_queue.cc:260] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.183622 1188 ts_tablet_manager.cc:1428] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.183900 1188 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.185102 1188 tablet_bootstrap.cc:654] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:31.758163 2424 rpcz_store.cc:276] 0110 02:51:25.172699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:25.172749 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:31.043411 (+5870662us) raft_consensus.cc:1466] Updating replica for 100 ops
0110 02:51:31.123839 (+ 80428us) raft_consensus.cc:1529] Early marking committed up to index 0
0110 02:51:31.123848 (+ 9us) raft_consensus.cc:1534] Triggering prepare for 100 ops
0110 02:51:31.124756 (+ 908us) write_op.cc:270] Start()
0110 02:51:31.124825 (+ 69us) write_op.cc:276] Timestamp: P: 1736477480020918 usec, L: 0
0110 02:51:31.138900 (+ 14075us) write_op.cc:270] Start()
0110 02:51:31.138946 (+ 46us) write_op.cc:276] Timestamp: P: 1736477480032064 usec, L: 0
0110 02:51:31.139218 (+ 272us) write_op.cc:270] Start()
0110 02:51:31.139259 (+ 41us) write_op.cc:276] Timestamp: P: 1736477480039829 usec, L: 0
0110 02:51:31.139425 (+ 166us) write_op.cc:270] Start()
0110 02:51:31.139449 (+ 24us) write_op.cc:276] Timestamp: P: 1736477480040658 usec, L: 0
0110 02:51:31.139656 (+ 207us) write_op.cc:270] Start()
0110 02:51:31.139679 (+ 23us) write_op.cc:276] Timestamp: P: 1736477480041524 usec, L: 0
0110 02:51:31.139826 (+ 147us) write_op.cc:270] Start()
0110 02:51:31.139848 (+ 22us) write_op.cc:276] Timestamp: P: 1736477480042271 usec, L: 0
0110 02:51:31.140022 (+ 174us) write_op.cc:270] Start()
0110 02:51:31.140081 (+ 59us) write_op.cc:276] Timestamp: P: 1736477480043028 usec, L: 0
0110 02:51:31.140241 (+ 160us) write_op.cc:270] Start()
0110 02:51:31.140266 (+ 25us) write_op.cc:276] Timestamp: P: 1736477480043800 usec, L: 0
0110 02:51:31.140390 (+ 124us) write_op.cc:270] Start()
0110 02:51:31.140408 (+ 18us) write_op.cc:276] Timestamp: P: 1736477480044493 usec, L: 0
0110 02:51:31.140543 (+ 135us) write_op.cc:270] Start()
0110 02:51:31.140563 (+ 20us) write_op.cc:276] Timestamp: P: 1736477480045282 usec, L: 0
0110 02:51:31.140723 (+ 160us) write_op.cc:270] Start()
0110 02:51:31.140743 (+ 20us) write_op.cc:276] Timestamp: P: 1736477480289844 usec, L: 0
0110 02:51:31.140888 (+ 145us) write_op.cc:270] Start()
0110 02:51:31.140911 (+ 23us) write_op.cc:276] Timestamp: P: 1736477480290525 usec, L: 0
0110 02:51:31.141092 (+ 181us) write_op.cc:270] Start()
0110 02:51:31.141110 (+ 18us) write_op.cc:276] Timestamp: P: 1736477480405096 usec, L: 0
0110 02:51:31.141271 (+ 161us) write_op.cc:270] Start()
0110 02:51:31.141310 (+ 39us) write_op.cc:276] Timestamp: P: 1736477480406269 usec, L: 0
0110 02:51:31.141479 (+ 169us) write_op.cc:270] Start()
0110 02:51:31.141508 (+ 29us) write_op.cc:276] Timestamp: P: 1736477480406892 usec, L: 0
0110 02:51:31.141714 (+ 206us) write_op.cc:270] Start()
0110 02:51:31.141733 (+ 19us) write_op.cc:276] Timestamp: P: 1736477480407460 usec, L: 0
0110 02:51:31.141914 (+ 181us) write_op.cc:270] Start()
0110 02:51:31.141951 (+ 37us) write_op.cc:276] Timestamp: P: 1736477480408079 usec, L: 0
0110 02:51:31.142144 (+ 193us) write_op.cc:270] Start()
0110 02:51:31.142170 (+ 26us) write_op.cc:276] Timestamp: P: 1736477480409574 usec, L: 0
0110 02:51:31.168769 (+ 26599us) write_op.cc:270] Start()
0110 02:51:31.168802 (+ 33us) write_op.cc:276] Timestamp: P: 1736477480410082 usec, L: 0
0110 02:51:31.168936 (+ 134us) write_op.cc:270] Start()
0110 02:51:31.168964 (+ 28us) write_op.cc:276] Timestamp: P: 1736477480410656 usec, L: 0
0110 02:51:31.169132 (+ 168us) write_op.cc:270] Start()
0110 02:51:31.169152 (+ 20us) write_op.cc:276] Timestamp: P: 1736477480411239 usec, L: 0
0110 02:51:31.169278 (+ 126us) write_op.cc:270] Start()
0110 02:51:31.169295 (+ 17us) write_op.cc:276] Timestamp: P: 1736477480434068 usec, L: 0
0110 02:51:31.169399 (+ 104us) write_op.cc:270] Start()
0110 02:51:31.169415 (+ 16us) write_op.cc:276] Timestamp: P: 1736477480434681 usec, L: 0
0110 02:51:31.169563 (+ 148us) write_op.cc:270] Start()
0110 02:51:31.169580 (+ 17us) write_op.cc:276] Timestamp: P: 1736477480435137 usec, L: 0
0110 02:51:31.169720 (+ 140us) write_op.cc:270] Start()
0110 02:51:31.169736 (+ 16us) write_op.cc:276] Timestamp: P: 1736477480484292 usec, L: 0
0110 02:51:31.169833 (+ 97us) write_op.cc:270] Start()
0110 02:51:31.169847 (+ 14us) write_op.cc:276] Timestamp: P: 1736477480484852 usec, L: 0
0110 02:51:31.169947 (+ 100us) write_op.cc:270] Start()
0110 02:51:31.169962 (+ 15us) write_op.cc:276] Timestamp: P: 1736477480487194 usec, L: 0
0110 02:51:31.170053 (+ 91us) write_op.cc:270] Start()
0110 02:51:31.170067 (+ 14us) write_op.cc:276] Timestamp: P: 1736477480508812 usec, L: 0
0110 02:51:31.170194 (+ 127us) write_op.cc:270] Start()
0110 02:51:31.170211 (+ 17us) write_op.cc:276] Timestamp: P: 1736477480514687 usec, L: 0
0110 02:51:31.170326 (+ 115us) write_op.cc:270] Start()
0110 02:51:31.170343 (+ 17us) write_op.cc:276] Timestamp: P: 1736477480534258 usec, L: 0
0110 02:51:31.170493 (+ 150us) write_op.cc:270] Start()
0110 02:51:31.170511 (+ 18us) write_op.cc:276] Timestamp: P: 1736477480539174 usec, L: 0
0110 02:51:31.170631 (+ 120us) write_op.cc:270] Start()
0110 02:51:31.170661 (+ 30us) write_op.cc:276] Timestamp: P: 1736477480662790 usec, L: 0
0110 02:51:31.170857 (+ 196us) write_op.cc:270] Start()
0110 02:51:31.170877 (+ 20us) write_op.cc:276] Timestamp: P: 1736477480702671 usec, L: 0
0110 02:51:31.171016 (+ 139us) write_op.cc:270] Start()
0110 02:51:31.171037 (+ 21us) write_op.cc:276] Timestamp: P: 1736477481420077 usec, L: 0
0110 02:51:31.171174 (+ 137us) write_op.cc:270] Start()
0110 02:51:31.171194 (+ 20us) write_op.cc:276] Timestamp: P: 1736477481450946 usec, L: 0
0110 02:51:31.171308 (+ 114us) write_op.cc:270] Start()
0110 02:51:31.171326 (+ 18us) write_op.cc:276] Timestamp: P: 1736477481452248 usec, L: 0
0110 02:51:31.171439 (+ 113us) write_op.cc:270] Start()
0110 02:51:31.171466 (+ 27us) write_op.cc:276] Timestamp: P: 1736477481453165 usec, L: 0
0110 02:51:31.171598 (+ 132us) write_op.cc:270] Start()
0110 02:51:31.171616 (+ 18us) write_op.cc:276] Timestamp: P: 1736477481453915 usec, L: 0
0110 02:51:31.171739 (+ 123us) write_op.cc:270] Start()
0110 02:51:31.171755 (+ 16us) write_op.cc:276] Timestamp: P: 1736477481454748 usec, L: 0
0110 02:51:31.171880 (+ 125us) write_op.cc:270] Start()
0110 02:51:31.171900 (+ 20us) write_op.cc:276] Timestamp: P: 1736477481493540 usec, L: 0
0110 02:51:31.172031 (+ 131us) write_op.cc:270] Start()
0110 02:51:31.172049 (+ 18us) write_op.cc:276] Timestamp: P: 1736477481494659 usec, L: 0
0110 02:51:31.172187 (+ 138us) write_op.cc:270] Start()
0110 02:51:31.172208 (+ 21us) write_op.cc:276] Timestamp: P: 1736477481610411 usec, L: 0
0110 02:51:31.172336 (+ 128us) write_op.cc:270] Start()
0110 02:51:31.172353 (+ 17us) write_op.cc:276] Timestamp: P: 1736477481651087 usec, L: 0
0110 02:51:31.172466 (+ 113us) write_op.cc:270] Start()
0110 02:51:31.172493 (+ 27us) write_op.cc:276] Timestamp: P: 1736477481662118 usec, L: 0
0110 02:51:31.172616 (+ 123us) write_op.cc:270] Start()
0110 02:51:31.172636 (+ 20us) write_op.cc:276] Timestamp: P: 1736477481681455 usec, L: 0
0110 02:51:31.312853 (+140217us) write_op.cc:270] Start()
0110 02:51:31.312901 (+ 48us) write_op.cc:276] Timestamp: P: 1736477481712977 usec, L: 0
0110 02:51:31.313104 (+ 203us) write_op.cc:270] Start()
0110 02:51:31.313131 (+ 27us) write_op.cc:276] Timestamp: P: 1736477481736440 usec, L: 0
0110 02:51:31.313304 (+ 173us) write_op.cc:270] Start()
0110 02:51:31.313328 (+ 24us) write_op.cc:276] Timestamp: P: 1736477481823431 usec, L: 0
0110 02:51:31.313468 (+ 140us) write_op.cc:270] Start()
0110 02:51:31.313492 (+ 24us) write_op.cc:276] Timestamp: P: 1736477481862673 usec, L: 0
0110 02:51:31.313700 (+ 208us) write_op.cc:270] Start()
0110 02:51:31.313739 (+ 39us) write_op.cc:276] Timestamp: P: 1736477481871441 usec, L: 0
0110 02:51:31.313968 (+ 229us) write_op.cc:270] Start()
0110 02:51:31.313990 (+ 22us) write_op.cc:276] Timestamp: P: 1736477481993481 usec, L: 0
0110 02:51:31.314126 (+ 136us) write_op.cc:270] Start()
0110 02:51:31.314143 (+ 17us) write_op.cc:276] Timestamp: P: 1736477482002426 usec, L: 0
0110 02:51:31.314278 (+ 135us) write_op.cc:270] Start()
0110 02:51:31.314299 (+ 21us) write_op.cc:276] Timestamp: P: 1736477482020283 usec, L: 0
0110 02:51:31.314441 (+ 142us) write_op.cc:270] Start()
0110 02:51:31.314462 (+ 21us) write_op.cc:276] Timestamp: P: 1736477482294062 usec, L: 0
0110 02:51:31.314597 (+ 135us) write_op.cc:270] Start()
0110 02:51:31.314617 (+ 20us) write_op.cc:276] Timestamp: P: 1736477482295972 usec, L: 0
0110 02:51:31.314746 (+ 129us) write_op.cc:270] Start()
0110 02:51:31.314766 (+ 20us) write_op.cc:276] Timestamp: P: 1736477482581450 usec, L: 0
0110 02:51:31.314943 (+ 177us) write_op.cc:270] Start()
0110 02:51:31.314965 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482602478 usec, L: 0
0110 02:51:31.315102 (+ 137us) write_op.cc:270] Start()
0110 02:51:31.315122 (+ 20us) write_op.cc:276] Timestamp: P: 1736477482611742 usec, L: 0
0110 02:51:31.315251 (+ 129us) write_op.cc:270] Start()
0110 02:51:31.315271 (+ 20us) write_op.cc:276] Timestamp: P: 1736477482631658 usec, L: 0
0110 02:51:31.315400 (+ 129us) write_op.cc:270] Start()
0110 02:51:31.315443 (+ 43us) write_op.cc:276] Timestamp: P: 1736477482653141 usec, L: 0
0110 02:51:31.315571 (+ 128us) write_op.cc:270] Start()
0110 02:51:31.315590 (+ 19us) write_op.cc:276] Timestamp: P: 1736477482738420 usec, L: 0
0110 02:51:31.315721 (+ 131us) write_op.cc:270] Start()
0110 02:51:31.315741 (+ 20us) write_op.cc:276] Timestamp: P: 1736477482761526 usec, L: 0
0110 02:51:31.315936 (+ 195us) write_op.cc:270] Start()
0110 02:51:31.315958 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482795498 usec, L: 0
0110 02:51:31.316118 (+ 160us) write_op.cc:270] Start()
0110 02:51:31.316139 (+ 21us) write_op.cc:276] Timestamp: P: 1736477482806237 usec, L: 0
0110 02:51:31.316298 (+ 159us) write_op.cc:270] Start()
0110 02:51:31.316320 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482829851 usec, L: 0
0110 02:51:31.316502 (+ 182us) write_op.cc:270] Start()
0110 02:51:31.316524 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482871094 usec, L: 0
0110 02:51:31.316664 (+ 140us) write_op.cc:270] Start()
0110 02:51:31.316686 (+ 22us) write_op.cc:276] Timestamp: P: 1736477482903008 usec, L: 0
0110 02:51:31.316814 (+ 128us) write_op.cc:270] Start()
0110 02:51:31.316834 (+ 20us) write_op.cc:276] Timestamp: P: 1736477483029100 usec, L: 0
0110 02:51:31.317024 (+ 190us) write_op.cc:270] Start()
0110 02:51:31.317057 (+ 33us) write_op.cc:276] Timestamp: P: 1736477483030443 usec, L: 0
0110 02:51:31.317203 (+ 146us) write_op.cc:270] Start()
0110 02:51:31.317223 (+ 20us) write_op.cc:276] Timestamp: P: 1736477483041904 usec, L: 0
0110 02:51:31.317359 (+ 136us) write_op.cc:270] Start()
0110 02:51:31.317379 (+ 20us) write_op.cc:276] Timestamp: P: 1736477483065508 usec, L: 0
0110 02:51:31.317529 (+ 150us) write_op.cc:270] Start()
0110 02:51:31.317548 (+ 19us) write_op.cc:276] Timestamp: P: 1736477483266625 usec, L: 0
0110 02:51:31.394165 (+ 76617us) write_op.cc:270] Start()
0110 02:51:31.394196 (+ 31us) write_op.cc:276] Timestamp: P: 1736477483295146 usec, L: 0
0110 02:51:31.394410 (+ 214us) write_op.cc:270] Start()
0110 02:51:31.394436 (+ 26us) write_op.cc:276] Timestamp: P: 1736477483396871 usec, L: 0
0110 02:51:31.394586 (+ 150us) write_op.cc:270] Start()
0110 02:51:31.394604 (+ 18us) write_op.cc:276] Timestamp: P: 1736477483413635 usec, L: 0
0110 02:51:31.394714 (+ 110us) write_op.cc:270] Start()
0110 02:51:31.394729 (+ 15us) write_op.cc:276] Timestamp: P: 1736477483571416 usec, L: 0
0110 02:51:31.394836 (+ 107us) write_op.cc:270] Start()
0110 02:51:31.394850 (+ 14us) write_op.cc:276] Timestamp: P: 1736477483600809 usec, L: 0
0110 02:51:31.394985 (+ 135us) write_op.cc:270] Start()
0110 02:51:31.395002 (+ 17us) write_op.cc:276] Timestamp: P: 1736477483662225 usec, L: 0
0110 02:51:31.395149 (+ 147us) write_op.cc:270] Start()
0110 02:51:31.395192 (+ 43us) write_op.cc:276] Timestamp: P: 1736477483881132 usec, L: 0
0110 02:51:31.395335 (+ 143us) write_op.cc:270] Start()
0110 02:51:31.395355 (+ 20us) write_op.cc:276] Timestamp: P: 1736477483930064 usec, L: 0
0110 02:51:31.395478 (+ 123us) write_op.cc:270] Start()
0110 02:51:31.395493 (+ 15us) write_op.cc:276] Timestamp: P: 1736477483950033 usec, L: 0
0110 02:51:31.395604 (+ 111us) write_op.cc:270] Start()
0110 02:51:31.395622 (+ 18us) write_op.cc:276] Timestamp: P: 1736477484089422 usec, L: 0
0110 02:51:31.395740 (+ 118us) write_op.cc:270] Start()
0110 02:51:31.395757 (+ 17us) write_op.cc:276] Timestamp: P: 1736477484119451 usec, L: 0
0110 02:51:31.395922 (+ 165us) write_op.cc:270] Start()
0110 02:51:31.395940 (+ 18us) write_op.cc:276] Timestamp: P: 1736477484205737 usec, L: 0
0110 02:51:31.396089 (+ 149us) write_op.cc:270] Start()
0110 02:51:31.396109 (+ 20us) write_op.cc:276] Timestamp: P: 1736477484219605 usec, L: 0
0110 02:51:31.545038 (+148929us) write_op.cc:270] Start()
0110 02:51:31.545080 (+ 42us) write_op.cc:276] Timestamp: P: 1736477484307386 usec, L: 0
0110 02:51:31.545255 (+ 175us) write_op.cc:270] Start()
0110 02:51:31.545271 (+ 16us) write_op.cc:276] Timestamp: P: 1736477484514827 usec, L: 0
0110 02:51:31.545378 (+ 107us) write_op.cc:270] Start()
0110 02:51:31.545394 (+ 16us) write_op.cc:276] Timestamp: P: 1736477484539111 usec, L: 0
0110 02:51:31.545502 (+ 108us) write_op.cc:270] Start()
0110 02:51:31.545517 (+ 15us) write_op.cc:276] Timestamp: P: 1736477484546192 usec, L: 0
0110 02:51:31.545610 (+ 93us) write_op.cc:270] Start()
0110 02:51:31.545625 (+ 15us) write_op.cc:276] Timestamp: P: 1736477484553123 usec, L: 0
0110 02:51:31.545732 (+ 107us) write_op.cc:270] Start()
0110 02:51:31.545747 (+ 15us) write_op.cc:276] Timestamp: P: 1736477484553715 usec, L: 0
0110 02:51:31.545847 (+ 100us) write_op.cc:270] Start()
0110 02:51:31.545862 (+ 15us) write_op.cc:276] Timestamp: P: 1736477484555675 usec, L: 0
0110 02:51:31.545983 (+ 121us) write_op.cc:270] Start()
0110 02:51:31.546000 (+ 17us) write_op.cc:276] Timestamp: P: 1736477485022601 usec, L: 0
0110 02:51:31.546104 (+ 104us) write_op.cc:270] Start()
0110 02:51:31.546119 (+ 15us) write_op.cc:276] Timestamp: P: 1736477485023249 usec, L: 0
0110 02:51:31.546221 (+ 102us) write_op.cc:270] Start()
0110 02:51:31.546237 (+ 16us) write_op.cc:276] Timestamp: P: 1736477485055389 usec, L: 0
0110 02:51:31.546364 (+ 127us) write_op.cc:270] Start()
0110 02:51:31.546403 (+ 39us) write_op.cc:276] Timestamp: P: 1736477485056614 usec, L: 0
0110 02:51:31.546535 (+ 132us) write_op.cc:270] Start()
0110 02:51:31.546551 (+ 16us) write_op.cc:276] Timestamp: P: 1736477485057509 usec, L: 0
0110 02:51:31.546650 (+ 99us) write_op.cc:270] Start()
0110 02:51:31.546666 (+ 16us) write_op.cc:276] Timestamp: P: 1736477485060166 usec, L: 0
0110 02:51:31.546801 (+ 135us) write_op.cc:270] Start()
0110 02:51:31.546817 (+ 16us) write_op.cc:276] Timestamp: P: 1736477485061385 usec, L: 0
0110 02:51:31.657483 (+110666us) log.cc:844] Serialized 68994 byte log entry
0110 02:51:31.691790 (+ 34307us) raft_consensus.cc:1640] Marking committed up to 741
0110 02:51:31.692040 (+ 250us) log.cc:844] Serialized 15 byte log entry
0110 02:51:31.723946 (+ 31906us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.723951 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:31.732386 (+ 8435us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.732392 (+ 6us) write_op.cc:454] Released schema lock
0110 02:51:31.754189 (+ 21797us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.754197 (+ 8us) write_op.cc:454] Released schema lock
0110 02:51:31.755572 (+ 1375us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.755579 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:31.757617 (+ 2038us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 02:51:31.757669 (+ 52us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 02:51:31.757798 (+ 129us) raft_consensus.cc:1680] finished
0110 02:51:31.757804 (+ 6us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 02:51:31.757968 (+ 164us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.137657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.137943 (+ 286us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.137952 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:31.137956 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.138202 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.138225 (+ 23us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.138232 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.138336 (+ 104us) tablet.cc:698] Row locks acquired
0110 02:51:31.138340 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.698660 (+560320us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.705608 (+ 6948us) write_op.cc:301] APPLY: starting
0110 02:51:31.705629 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.705914 (+ 285us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.705917 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.706062 (+ 145us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.706282 (+ 220us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.706322 (+ 40us) write_op.cc:312] APPLY: finished
0110 02:51:31.706547 (+ 225us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.706742 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.724173 (+ 17431us) write_op.cc:454] Released schema lock
0110 02:51:31.724180 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.724230 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.724235 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:31.159071 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.159215 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.159225 (+ 10us) write_op.cc:435] Acquired schema lock
0110 02:51:31.159227 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.159328 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.159368 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.159373 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.159459 (+ 86us) tablet.cc:698] Row locks acquired
0110 02:51:31.159462 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.708867 (+549405us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.723147 (+ 14280us) write_op.cc:301] APPLY: starting
0110 02:51:31.723163 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.723207 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.723210 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.723372 (+ 162us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.723406 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.723464 (+ 58us) write_op.cc:312] APPLY: finished
0110 02:51:31.723585 (+ 121us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.723670 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.723690 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:31.723706 (+ 16us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:31.159531 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.159618 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.159623 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.159625 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.159683 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.159692 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.159696 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.159741 (+ 45us) tablet.cc:698] Row locks acquired
0110 02:51:31.159743 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.724084 (+564341us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.724306 (+ 222us) write_op.cc:301] APPLY: starting
0110 02:51:31.724321 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.724362 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.724366 (+ 4us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.724478 (+ 112us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.724513 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.724553 (+ 40us) write_op.cc:312] APPLY: finished
0110 02:51:31.724621 (+ 68us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.724684 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.724699 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:31.724706 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.724759 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.724764 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:31.159786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.159840 (+ 54us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.159843 (+ 3us) write_op.cc:435] Acquired schema lock
0110 02:51:31.159845 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.159890 (+ 45us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.159897 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.159900 (+ 3us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.159954 (+ 54us) tablet.cc:698] Row locks acquired
0110 02:51:31.159956 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.724153 (+564197us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.724824 (+ 671us) write_op.cc:301] APPLY: starting
0110 02:51:31.724831 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.724855 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.724857 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.724971 (+ 114us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.725002 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.725033 (+ 31us) write_op.cc:312] APPLY: finished
0110 02:51:31.725085 (+ 52us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.725132 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.725151 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:31.725157 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:31.160005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.160059 (+ 54us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.160062 (+ 3us) write_op.cc:435] Acquired schema lock
0110 02:51:31.160064 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.160129 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.160139 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.160143 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.160188 (+ 45us) tablet.cc:698] Row locks acquired
0110 02:51:31.160190 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.732465 (+572275us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.733068 (+ 603us) write_op.cc:301] APPLY: starting
0110 02:51:31.733084 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.733116 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.733118 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.733203 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.733230 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:31.733252 (+ 22us) write_op.cc:312] APPLY: finished
0110 02:51:31.733311 (+ 59us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.733355 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.733367 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:31.733372 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.733414 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.733416 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:31.160238 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.160303 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.160307 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:31.160309 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.160379 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.160388 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.160392 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.160439 (+ 47us) tablet.cc:698] Row locks acquired
0110 02:51:31.160442 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.732520 (+572078us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.733471 (+ 951us) write_op.cc:301] APPLY: starting
0110 02:51:31.733479 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.733503 (+ 24us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.733505 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.733573 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.733635 (+ 62us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.733658 (+ 23us) write_op.cc:312] APPLY: finished
0110 02:51:31.733705 (+ 47us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.733763 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.742284 (+ 8521us) write_op.cc:454] Released schema lock
0110 02:51:31.742291 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.742341 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.742344 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:31.160508 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.160580 (+ 72us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.160585 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.160587 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.160656 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.160664 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.160669 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:31.160709 (+ 40us) tablet.cc:698] Row locks acquired
0110 02:51:31.160711 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.732553 (+571842us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.742418 (+ 9865us) write_op.cc:301] APPLY: starting
0110 02:51:31.742429 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.742456 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.742458 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.742497 (+ 39us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.742518 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.742543 (+ 25us) write_op.cc:312] APPLY: finished
0110 02:51:31.742592 (+ 49us) log.cc:844] Serialized 23 byte log entry
0110 02:51:31.742635 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.742645 (+ 10us) write_op.cc:454] Released schema lock
0110 02:51:31.742650 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.742689 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.742692 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:31.160767 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.160846 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.160850 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:31.160852 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.160909 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.160918 (+ 9us) write_op
I20250110 02:51:32.195366 2122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5d8d40556dc348e8aa8255d116251a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } } }
I20250110 02:51:32.195598 2122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10 [sys.catalog]: This master's current role is: LEADER
I20250110 02:51:32.173808 1184 tablet_bootstrap.cc:492] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.198546 1184 ts_tablet_manager.cc:1397] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.001s
I20250110 02:51:32.199486 1184 raft_consensus.cc:357] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.199679 1184 raft_consensus.cc:383] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.199739 1184 raft_consensus.cc:738] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.199951 1184 consensus_queue.cc:260] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.204478 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 305) took 4745 ms. Trace:
I20250110 02:51:32.204571 2224 rpcz_store.cc:276] 0110 02:51:27.459274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.459431 (+ 157us) service_pool.cc:225] Handling call
0110 02:51:32.204349 (+4744918us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:32.204360 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.204439 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4744774}
W20250110 02:51:32.204998 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d9018955f3884824beb2e7336987fc1c because the tablet was not found. No further retry: Not found: Tablet not found: d9018955f3884824beb2e7336987fc1c
I20250110 02:51:32.207130 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 306) took 4747 ms. Trace:
I20250110 02:51:32.207221 2224 rpcz_store.cc:276] 0110 02:51:27.459685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.459853 (+ 168us) service_pool.cc:225] Handling call
0110 02:51:32.207044 (+4747191us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:32.207053 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.207112 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4747047}
W20250110 02:51:32.207619 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 7d4422eca5604d149e73c2a669d07af0 because the tablet was not found. No further retry: Not found: Tablet not found: 7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:32.208482 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 307) took 4748 ms. Trace:
I20250110 02:51:32.208710 2224 rpcz_store.cc:276] 0110 02:51:27.460147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.460340 (+ 193us) service_pool.cc:225] Handling call
0110 02:51:32.208409 (+4748069us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:32.208417 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.208467 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4747930}
W20250110 02:51:32.209255 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because the tablet was not found. No further retry: Not found: Tablet not found: f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:32.209728 908 raft_consensus.cc:3054] T 39cfc508c1b84299a2f16d307bb25354 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.211900 2352 ts_tablet_manager.cc:1918] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.211988 2352 log.cc:1198] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/a81c031451104d5a93a1672b4b1918c5
I20250110 02:51:32.212265 2352 ts_tablet_manager.cc:1939] T a81c031451104d5a93a1672b4b1918c5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.213068 2224 tablet_replica.cc:331] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.214124 2224 raft_consensus.cc:2238] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.214332 2224 raft_consensus.cc:2267] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.216681 2905 raft_consensus.cc:491] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.217196 1181 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.220942 1181 ts_tablet_manager.cc:1397] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.003s
I20250110 02:51:32.220891 2905 raft_consensus.cc:513] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.221647 2905 leader_election.cc:290] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.221621 1181 raft_consensus.cc:357] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.221933 1181 raft_consensus.cc:383] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.221997 1181 raft_consensus.cc:738] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.222221 1181 consensus_queue.cc:260] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.222321 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.175182 2357 rpcz_store.cc:276] 0110 02:51:27.530789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.530826 (+ 37us) service_pool.cc:225] Handling call
0110 02:51:32.175035 (+4644209us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:32.175043 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.175096 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4644017}
I20250110 02:51:32.222640 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.222744 1181 ts_tablet_manager.cc:1428] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:32.222836 774 raft_consensus.cc:2463] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:32.222910 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because the tablet was not found. No further retry: Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:32.223011 1181 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.223289 996 leader_election.cc:304] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.223863 1188 tablet_bootstrap.cc:492] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.223923 2224 ts_tablet_manager.cc:1905] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.223992 1188 ts_tablet_manager.cc:1397] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250110 02:51:32.224581 1188 raft_consensus.cc:357] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.224764 1188 raft_consensus.cc:383] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.224812 1188 raft_consensus.cc:738] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.224867 2898 raft_consensus.cc:2798] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.224958 2898 raft_consensus.cc:491] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.225020 2898 raft_consensus.cc:3054] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:32.226675 994 leader_election.cc:343] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.228027 2898 raft_consensus.cc:513] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.228600 2898 leader_election.cc:290] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.229116 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.229318 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5dab241cd74f329a57e6f19e95a7b6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.229483 774 raft_consensus.cc:3054] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:32.229605 994 leader_election.cc:343] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.231302 1181 tablet_bootstrap.cc:654] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.231976 774 raft_consensus.cc:2463] T ac5dab241cd74f329a57e6f19e95a7b6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.232460 996 leader_election.cc:304] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.232784 2456 raft_consensus.cc:2798] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.232910 2456 raft_consensus.cc:695] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.232985 2357 tablet_replica.cc:331] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.233173 2357 raft_consensus.cc:2238] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.233125 2456 consensus_queue.cc:237] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.233299 2357 raft_consensus.cc:2267] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.233793 2357 ts_tablet_manager.cc:1905] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.237493 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:32.239446 1184 ts_tablet_manager.cc:1428] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250110 02:51:32.239975 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:32.242707 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:32.242964 1188 consensus_queue.cc:260] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.245261 1188 ts_tablet_manager.cc:1428] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250110 02:51:32.245718 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:32.246505 1188 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.246670 1184 tablet_bootstrap.cc:492] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.252058 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:32.252604 2289 consensus_queue.cc:1035] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:32.253141 994 consensus_peers.cc:487] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.247737 914 raft_consensus.cc:3054] T 84adc1fd3f76497b98a0fcbc52a3120a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.253513 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:32.253683 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:32.262445 1449 catalog_manager.cc:5526] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.264142 1188 tablet_bootstrap.cc:654] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.264436 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:32.264741 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:32.265072 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:32.265278 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:32.265959 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:32.268103 2357 ts_tablet_manager.cc:1918] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.268337 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ce4fe51f2e914865a2e8bbaa256b8098 (table default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc]). Expected version 1 got 0
I20250110 02:51:32.268939 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:32.269209 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 815767a4c7624852bfedab9fa6cb5f44 (table default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462]). Expected version 1 got 0
I20250110 02:51:32.269517 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 75ca5369caf3415ea508352362409656 (table default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e]). Expected version 1 got 0
I20250110 02:51:32.269878 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 00ebf8debb9d4394839d1c1cb97820a5 (table default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5]). Expected version 2 got 0
I20250110 02:51:32.270484 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:32.272398 1181 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.272552 1181 ts_tablet_manager.cc:1397] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.007s
I20250110 02:51:32.273087 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 6c4bd5e8317a4801a8134d58c23b7bee (table default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f]). Expected version 1 got 0
I20250110 02:51:32.273464 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:32.273811 1181 raft_consensus.cc:357] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.274056 1181 raft_consensus.cc:383] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.274256 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:32.274644 1181 raft_consensus.cc:738] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.275230 1181 consensus_queue.cc:260] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.276381 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:32.278681 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:32.284091 2357 log.cc:1198] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/c28f191afa2a43608bbd74322bec97a6
I20250110 02:51:32.284621 2357 ts_tablet_manager.cc:1939] T c28f191afa2a43608bbd74322bec97a6 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.288766 1188 tablet_bootstrap.cc:492] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.288976 1188 ts_tablet_manager.cc:1397] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250110 02:51:32.288967 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 206) took 4757 ms. Trace:
I20250110 02:51:32.289050 2357 rpcz_store.cc:276] 0110 02:51:27.531350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.531387 (+ 37us) service_pool.cc:225] Handling call
0110 02:51:32.232935 (+4701548us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:32.232946 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.237079 (+ 4133us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.267994 (+ 30915us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.288949 (+ 20955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4701376}
I20250110 02:51:32.289675 1188 raft_consensus.cc:357] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.289950 1188 raft_consensus.cc:383] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.290055 1188 raft_consensus.cc:738] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.290277 1188 consensus_queue.cc:260] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.290943 1188 ts_tablet_manager.cc:1428] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.291211 1188 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.292611 1188 tablet_bootstrap.cc:654] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.292428 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet c28f191afa2a43608bbd74322bec97a6 (table default.Table_82157dfbcfe64e759c6847722284ee3f [id=c6f818c8ec264520a7f85dd60ab6ce1d]) successfully deleted
I20250110 02:51:32.293826 1184 tablet_bootstrap.cc:654] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.240850 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 233) took 5922 ms. Trace:
I20250110 02:51:32.251585 2691 raft_consensus.cc:491] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.294032 2691 raft_consensus.cc:513] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.294651 2691 leader_election.cc:290] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.294850 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:32.294796 2352 rpcz_store.cc:276] 0110 02:51:26.318299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.318431 (+ 132us) service_pool.cc:225] Handling call
0110 02:51:32.011650 (+5693219us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:32.011659 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.139893 (+128234us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.211794 (+ 71901us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.240825 (+ 29031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5692880}
I20250110 02:51:32.295147 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 234) took 5976 ms. Trace:
I20250110 02:51:32.295399 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet a81c031451104d5a93a1672b4b1918c5 (table default.Table_e7d7ae7a9eea4238802be676c6f59905 [id=0c92149e75c8456484ef2e8a5ab7f772]) successfully deleted
I20250110 02:51:32.295538 2352 rpcz_store.cc:276] 0110 02:51:26.318552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.318794 (+ 242us) service_pool.cc:225] Handling call
0110 02:51:32.295056 (+5976262us) ts_tablet_manager.cc:1145] Deleting tablet 8e79175853474cbea951f5a5686bf630
0110 02:51:32.295065 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.295133 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5976154}
W20250110 02:51:32.296398 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because the tablet was not found. No further retry: Not found: Tablet not found: 8e79175853474cbea951f5a5686bf630
I20250110 02:51:32.275396 2898 consensus_queue.cc:1035] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.295646 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:32.296828 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:32.296875 1181 ts_tablet_manager.cc:1428] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.024s user 0.000s sys 0.001s
I20250110 02:51:32.297142 1181 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.297298 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
W20250110 02:51:32.283234 994 consensus_peers.cc:487] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.285049 2224 ts_tablet_manager.cc:1918] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.295099 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.303465 2224 log.cc:1198] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/dba0cb7970f1453dac256b255ed7cac0
I20250110 02:51:32.303484 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4f1d7940a1574af1804d5f5139ca7494 (table default.Table_b4223b31a5974abb9725bceb8026a8ae [id=4e2fb95ceb8c43fc920b1cd9e73557df]). Expected version 1 got 0
I20250110 02:51:32.304702 2224 ts_tablet_manager.cc:1939] T dba0cb7970f1453dac256b255ed7cac0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.304950 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:32.306484 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 308) took 4845 ms. Trace:
I20250110 02:51:32.307819 2224 rpcz_store.cc:276] 0110 02:51:27.460588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.460737 (+ 149us) service_pool.cc:225] Handling call
0110 02:51:32.213028 (+4752291us) ts_tablet_manager.cc:1145] Deleting tablet dba0cb7970f1453dac256b255ed7cac0
0110 02:51:32.213036 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.265755 (+ 52719us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.284948 (+ 19193us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.306468 (+ 21520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4752145}
I20250110 02:51:32.308276 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet dba0cb7970f1453dac256b255ed7cac0 (table default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]) successfully deleted
I20250110 02:51:32.308588 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 235) took 5989 ms. Trace:
I20250110 02:51:32.308689 2352 rpcz_store.cc:276] 0110 02:51:26.319111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.319152 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.296406 (+5977254us) ts_tablet_manager.cc:1145] Deleting tablet d4f1d916a5ea48708c2c1a61c92fe137
0110 02:51:32.296415 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.308567 (+ 12152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5977132}
I20250110 02:51:32.309311 2352 tablet_replica.cc:331] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.309317 2224 tablet_replica.cc:331] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.309484 2352 raft_consensus.cc:2238] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.294468 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 207) took 4763 ms. Trace:
I20250110 02:51:32.309690 2224 raft_consensus.cc:2238] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.309770 2357 rpcz_store.cc:276] 0110 02:51:27.531454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.531493 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:32.292598 (+4761105us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:32.292609 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.294453 (+ 1844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4760916}
I20250110 02:51:32.310128 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 208) took 4778 ms. Trace:
I20250110 02:51:32.310218 2357 rpcz_store.cc:276] 0110 02:51:27.531559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.531595 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.310036 (+4778441us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:32.310045 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.310114 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4778302}
I20250110 02:51:32.310511 2224 raft_consensus.cc:2267] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.310556 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 209) took 4778 ms. Trace:
W20250110 02:51:32.310575 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because the tablet was not found. No further retry: Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137
I20250110 02:51:32.310660 2357 rpcz_store.cc:276] 0110 02:51:27.531674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.531714 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:32.310464 (+4778750us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:32.310472 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.310543 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4778594}
I20250110 02:51:32.310912 2357 tablet_replica.cc:331] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.295717 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f74fe375440a28139c27f3f868970" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
W20250110 02:51:32.311369 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f5e88d1bbc304b4ebbc849a7d783052e because the tablet was not found. No further retry: Not found: Tablet not found: f5e88d1bbc304b4ebbc849a7d783052e
W20250110 02:51:32.311614 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because the tablet was not found. No further retry: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb
W20250110 02:51:32.311823 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:32.311843 2357 raft_consensus.cc:2238] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.311996 2357 raft_consensus.cc:2267] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.312088 853 leader_election.cc:304] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.312484 2691 raft_consensus.cc:2743] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.312717 2357 ts_tablet_manager.cc:1905] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.309613 2352 raft_consensus.cc:2267] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.319550 2352 ts_tablet_manager.cc:1905] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.320879 2224 ts_tablet_manager.cc:1905] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.321280 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:32.323297 1184 tablet_bootstrap.cc:492] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.323419 1184 ts_tablet_manager.cc:1397] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.000s
I20250110 02:51:32.324045 1184 raft_consensus.cc:357] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.324227 1184 raft_consensus.cc:383] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.322192 1181 tablet_bootstrap.cc:654] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.324290 1184 raft_consensus.cc:738] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.324470 1184 consensus_queue.cc:260] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.325048 1184 ts_tablet_manager.cc:1428] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:32.325301 1184 tablet_bootstrap.cc:492] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.326520 1184 tablet_bootstrap.cc:654] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.327147 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:32.327440 2905 consensus_queue.cc:1035] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.327977 2357 ts_tablet_manager.cc:1918] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.328065 2357 log.cc:1198] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/08705d260c0a462b94ba6b0dc5121123
I20250110 02:51:32.328385 2357 ts_tablet_manager.cc:1939] T 08705d260c0a462b94ba6b0dc5121123 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.329562 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:32.331148 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 181) took 4848 ms. Trace:
I20250110 02:51:32.331265 2357 rpcz_store.cc:276] 0110 02:51:27.482791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.482845 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:32.310876 (+4828031us) ts_tablet_manager.cc:1145] Deleting tablet 08705d260c0a462b94ba6b0dc5121123
0110 02:51:32.310883 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.315964 (+ 5081us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.327755 (+ 11791us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.331120 (+ 3365us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":725,"tablet-delete.queue_time_us":4774912}
I20250110 02:51:32.334323 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 182) took 4851 ms. Trace:
I20250110 02:51:32.334447 2357 rpcz_store.cc:276] 0110 02:51:27.483241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.483287 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:32.332130 (+4848843us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:32.332139 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.334301 (+ 2162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4796092}
I20250110 02:51:32.332351 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 08705d260c0a462b94ba6b0dc5121123 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:32.334753 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 183) took 4851 ms. Trace:
I20250110 02:51:32.334839 2357 rpcz_store.cc:276] 0110 02:51:27.483638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.483680 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:32.334668 (+4850988us) ts_tablet_manager.cc:1145] Deleting tablet 302af84fd8374ffa96a7503b54849536
0110 02:51:32.334676 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.334734 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4798570}
W20250110 02:51:32.334950 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because the tablet was not found. No further retry: Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:32.335165 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 184) took 4851 ms. Trace:
W20250110 02:51:32.335284 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because the tablet was not found. No further retry: Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:32.335247 2357 rpcz_store.cc:276] 0110 02:51:27.484151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.484395 (+ 244us) service_pool.cc:225] Handling call
0110 02:51:32.335097 (+4850702us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:32.335104 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.335151 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4798959}
I20250110 02:51:32.335567 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 185) took 4851 ms. Trace:
W20250110 02:51:32.335673 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because the tablet was not found. No further retry: Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:32.336105 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:32.337214 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:32.338449 2357 rpcz_store.cc:276] 0110 02:51:27.484244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.484526 (+ 282us) service_pool.cc:225] Handling call
0110 02:51:32.335477 (+4850951us) ts_tablet_manager.cc:1145] Deleting tablet b4f1782c46a44ed091f4c533226c71f2
0110 02:51:32.335484 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.335554 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4799274}
I20250110 02:51:32.338878 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 186) took 4854 ms. Trace:
I20250110 02:51:32.338973 2357 rpcz_store.cc:276] 0110 02:51:27.484353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.484638 (+ 285us) service_pool.cc:225] Handling call
0110 02:51:32.338783 (+4854145us) ts_tablet_manager.cc:1145] Deleting tablet 8d1f2acd39be40e4b71cc0dcf68e7567
0110 02:51:32.338791 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.338863 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4802532}
I20250110 02:51:32.339284 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 210) took 4786 ms. Trace:
I20250110 02:51:32.339387 2357 rpcz_store.cc:276] 0110 02:51:27.553157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.553205 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:32.339185 (+4785980us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:32.339195 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.339266 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4743922}
I20250110 02:51:32.339707 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 211) took 4777 ms. Trace:
I20250110 02:51:32.339803 2357 rpcz_store.cc:276] 0110 02:51:27.562408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.562535 (+ 127us) service_pool.cc:225] Handling call
0110 02:51:32.339636 (+4777101us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:32.339644 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.339691 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4733720}
I20250110 02:51:32.340077 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 215) took 4733 ms. Trace:
W20250110 02:51:32.338995 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
W20250110 02:51:32.341622 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:32.341945 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
W20250110 02:51:32.342262 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because the tablet was not found. No further retry: Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:32.342662 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:32.340157 2357 rpcz_store.cc:276] 0110 02:51:27.606549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.606602 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:32.340013 (+4733411us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:32.340021 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.340065 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4733237}
I20250110 02:51:32.343107 914 raft_consensus.cc:3054] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.343333 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 213) took 4737 ms. Trace:
W20250110 02:51:32.343449 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because the tablet was not found. No further retry: Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:32.343415 2357 rpcz_store.cc:276] 0110 02:51:27.606114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.606215 (+ 101us) service_pool.cc:225] Handling call
0110 02:51:32.343261 (+4737046us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:32.343270 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.343319 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4732700}
I20250110 02:51:32.343715 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 216) took 4737 ms. Trace:
I20250110 02:51:32.343792 2357 rpcz_store.cc:276] 0110 02:51:27.606627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.607018 (+ 391us) service_pool.cc:225] Handling call
0110 02:51:32.343646 (+4736628us) ts_tablet_manager.cc:1145] Deleting tablet 971f97a3be9b49958b1218a275e126d4
0110 02:51:32.343654 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.343702 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4732357}
I20250110 02:51:32.344054 2357 tablet_replica.cc:331] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
W20250110 02:51:32.343914 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because the tablet was not found. No further retry: Not found: Tablet not found: f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:32.344254 2357 raft_consensus.cc:2238] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.344427 2357 raft_consensus.cc:2267] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:32.344419 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 971f97a3be9b49958b1218a275e126d4 because the tablet was not found. No further retry: Not found: Tablet not found: 971f97a3be9b49958b1218a275e126d4
I20250110 02:51:32.345029 2357 ts_tablet_manager.cc:1905] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.345832 1181 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.346031 1181 ts_tablet_manager.cc:1397] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250110 02:51:32.346168 2456 consensus_queue.cc:1035] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.346323 2352 ts_tablet_manager.cc:1918] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.346522 2352 log.cc:1198] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d00698a0a9ee4e5da69a745f4a3aa8b4
I20250110 02:51:32.346844 2352 ts_tablet_manager.cc:1939] T d00698a0a9ee4e5da69a745f4a3aa8b4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.346871 1181 raft_consensus.cc:357] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.347052 1181 raft_consensus.cc:383] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.347095 1181 raft_consensus.cc:738] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.347245 1181 consensus_queue.cc:260] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.347568 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 236) took 6028 ms. Trace:
I20250110 02:51:32.347698 1181 ts_tablet_manager.cc:1428] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 02:51:32.347642 2352 rpcz_store.cc:276] 0110 02:51:26.319504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.319544 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:32.309266 (+5989722us) ts_tablet_manager.cc:1145] Deleting tablet d00698a0a9ee4e5da69a745f4a3aa8b4
0110 02:51:32.309275 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.333765 (+ 24490us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.346227 (+ 12462us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.347550 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5376,"tablet-delete.queue_time_us":5989594}
I20250110 02:51:32.347990 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d00698a0a9ee4e5da69a745f4a3aa8b4 (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]) successfully deleted
I20250110 02:51:32.348680 1184 tablet_bootstrap.cc:492] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.348784 1184 ts_tablet_manager.cc:1397] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.002s
I20250110 02:51:32.349393 1184 raft_consensus.cc:357] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.349565 1184 raft_consensus.cc:383] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.349614 1184 raft_consensus.cc:738] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.349992 1184 consensus_queue.cc:260] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.350575 1184 ts_tablet_manager.cc:1428] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:32.350862 1184 tablet_bootstrap.cc:492] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.352048 2352 tablet_replica.cc:331] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.352353 2352 raft_consensus.cc:2238] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.352483 1184 tablet_bootstrap.cc:654] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.352609 2352 raft_consensus.cc:2267] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.353299 2352 ts_tablet_manager.cc:1905] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.355234 911 raft_consensus.cc:3054] T c52f74fe375440a28139c27f3f868970 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.356014 1181 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.357689 1181 tablet_bootstrap.cc:654] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.362042 2357 ts_tablet_manager.cc:1918] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.344319 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:32.362596 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3aee71c51a96482fbb527576867bbaa9 (table default.Table_a99fe640359046aa8ff288db7a687676 [id=c9a367ed70c742a18d7dc76e72b37300]). Expected version 1 got 0
I20250110 02:51:32.362927 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 1 got 0
I20250110 02:51:32.365224 2357 log.cc:1198] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a7d4f03a8c66446ea7c6d178cc7ef80a
I20250110 02:51:32.365541 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_3993d1a3d35d4f9cb244a585f4d0084e [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 2 got 0
I20250110 02:51:32.362701 1184 tablet_bootstrap.cc:492] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.365839 1184 ts_tablet_manager.cc:1397] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250110 02:51:32.367133 2357 ts_tablet_manager.cc:1939] T a7d4f03a8c66446ea7c6d178cc7ef80a P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:32.367502 1444 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24cf81039d5e4a47b426ea5d2f21acc2 consensus state has no leader
I20250110 02:51:32.367672 1444 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 163) took 3433 ms. Trace:
I20250110 02:51:32.367939 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 212) took 4761 ms. Trace:
I20250110 02:51:32.368009 2357 rpcz_store.cc:276] 0110 02:51:27.605996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.606192 (+ 196us) service_pool.cc:225] Handling call
0110 02:51:32.344015 (+4737823us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:32.344022 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.354559 (+ 10537us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.361956 (+ 7397us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.367918 (+ 5962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4732570}
I20250110 02:51:32.367764 1444 rpcz_store.cc:276] 0110 02:51:28.934270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.252592 (+2318322us) service_pool.cc:225] Handling call
0110 02:51:31.257426 (+ 4834us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:31.257655 (+ 229us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:31.257826 (+ 171us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:31.257883 (+ 57us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:31.263730 (+ 5847us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:32.367640 (+1103910us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.258314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.258410 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.258414 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:31.258416 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.258518 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.258526 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.258531 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:31.258585 (+ 54us) tablet.cc:698] Row locks acquired
0110 02:51:31.258589 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.258621 (+ 32us) write_op.cc:270] Start()
0110 02:51:31.258649 (+ 28us) write_op.cc:276] Timestamp: P: 1736477491258574 usec, L: 0
0110 02:51:31.258652 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.258822 (+ 170us) log.cc:844] Serialized 405 byte log entry
0110 02:51:31.262777 (+ 3955us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.263015 (+ 238us) write_op.cc:301] APPLY: starting
0110 02:51:31.263027 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.263058 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.263078 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.263124 (+ 46us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.263149 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.263155 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:31.263208 (+ 53us) log.cc:844] Serialized 23 byte log entry
0110 02:51:31.263254 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.263267 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:31.263272 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.263330 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.263333 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":310,"apply.run_wall_time_us":310,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":568,"prepare.run_wall_time_us":569,"replication_time_us":4102}]]}
I20250110 02:51:32.368412 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a7d4f03a8c66446ea7c6d178cc7ef80a (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:32.366710 1184 raft_consensus.cc:357] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.368990 1184 raft_consensus.cc:383] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.369055 1184 raft_consensus.cc:738] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.369261 1184 consensus_queue.cc:260] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.369724 1184 ts_tablet_manager.cc:1428] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:32.370137 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 214) took 4763 ms. Trace:
I20250110 02:51:32.370213 2357 rpcz_store.cc:276] 0110 02:51:27.606420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.607429 (+ 1009us) service_pool.cc:225] Handling call
0110 02:51:32.370052 (+4762623us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:32.370060 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.370123 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4757844}
W20250110 02:51:32.370642 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because the tablet was not found. No further retry: Not found: Tablet not found: 747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:32.370898 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:39025
I20250110 02:51:32.371040 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:32.371485 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 217) took 4666 ms. Trace:
I20250110 02:51:32.371971 2352 ts_tablet_manager.cc:1918] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:32.372192 994 consensus_peers.cc:487] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.372128 1184 tablet_bootstrap.cc:492] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.372179 2357 rpcz_store.cc:276] 0110 02:51:27.704991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.705037 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:32.371407 (+4666370us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:32.371415 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.371470 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4616161}
W20250110 02:51:32.373853 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 32dd1e726f1a41d0894737b3ef3caf8f because the tablet was not found. No further retry: Not found: Tablet not found: 32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:32.374228 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 219) took 4229 ms. Trace:
I20250110 02:51:32.374404 2352 log.cc:1198] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/676f5f457d9e4028bfd0b707604c42f5
I20250110 02:51:32.374783 2352 ts_tablet_manager.cc:1939] T 676f5f457d9e4028bfd0b707604c42f5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.374871 2357 rpcz_store.cc:276] 0110 02:51:28.145113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.145176 (+ 63us) service_pool.cc:225] Handling call
0110 02:51:32.374122 (+4228946us) ts_tablet_manager.cc:1145] Deleting tablet dc6b2654f86e4686b8a3034de02d11bf
0110 02:51:32.374132 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.374212 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4228787}
W20250110 02:51:32.375342 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dc6b2654f86e4686b8a3034de02d11bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc6b2654f86e4686b8a3034de02d11bf already in progress: creating tablet
I20250110 02:51:32.375643 2357 tablet_replica.cc:331] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.376410 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 238) took 6056 ms. Trace:
I20250110 02:51:32.376678 2357 raft_consensus.cc:2238] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.377655 2357 raft_consensus.cc:2267] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:32.377812 996 consensus_peers.cc:487] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08705d260c0a462b94ba6b0dc5121123. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.378458 2357 ts_tablet_manager.cc:1905] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.378118 2352 rpcz_store.cc:276] 0110 02:51:26.320163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.320213 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:32.351988 (+6031775us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:32.352000 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.359816 (+ 7816us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.371869 (+ 12053us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.376392 (+ 4523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6031649}
I20250110 02:51:32.378911 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 676f5f457d9e4028bfd0b707604c42f5 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]) successfully deleted
I20250110 02:51:32.377521 1184 tablet_bootstrap.cc:654] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.380054 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 239) took 6059 ms. Trace:
I20250110 02:51:32.383531 2352 rpcz_store.cc:276] 0110 02:51:26.320480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.320516 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.379971 (+6059455us) ts_tablet_manager.cc:1145] Deleting tablet 3b6d57f3e9a44d178300d3505d504854
0110 02:51:32.379979 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.380040 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6059339}
W20250110 02:51:32.384008 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because the tablet was not found. No further retry: Not found: Tablet not found: 3b6d57f3e9a44d178300d3505d504854
I20250110 02:51:32.384228 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 240) took 6063 ms. Trace:
I20250110 02:51:32.384331 2352 rpcz_store.cc:276] 0110 02:51:26.320763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.320886 (+ 123us) service_pool.cc:225] Handling call
0110 02:51:32.384139 (+6063253us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:32.384153 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.384213 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6062953}
I20250110 02:51:32.384704 2352 tablet_replica.cc:331] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:32.384720 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because the tablet was not found. No further retry: Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:32.384922 2352 raft_consensus.cc:2238] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:32.385237 2352 raft_consensus.cc:2267] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.385721 2352 ts_tablet_manager.cc:1905] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.391386 2224 ts_tablet_manager.cc:1918] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.392077 1188 tablet_bootstrap.cc:492] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.392263 1188 ts_tablet_manager.cc:1397] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.000s
I20250110 02:51:32.392992 2224 log.cc:1198] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:32.392946 1188 raft_consensus.cc:357] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.393240 1188 raft_consensus.cc:383] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.393340 1188 raft_consensus.cc:738] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.393435 2224 ts_tablet_manager.cc:1939] T c00e97e496b54dcf82b7182591d6f4aa P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.393631 1188 consensus_queue.cc:260] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.394289 1188 ts_tablet_manager.cc:1428] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:32.394611 1188 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.395484 3105 consensus_queue.cc:1035] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.396690 1188 tablet_bootstrap.cc:654] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.399430 1184 tablet_bootstrap.cc:492] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.399698 1184 ts_tablet_manager.cc:1397] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250110 02:51:32.400285 1184 raft_consensus.cc:357] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.400483 1184 raft_consensus.cc:383] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.400547 1184 raft_consensus.cc:738] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.400734 1184 consensus_queue.cc:260] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.401182 1184 ts_tablet_manager.cc:1428] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:32.401435 1184 tablet_bootstrap.cc:492] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.402863 1184 tablet_bootstrap.cc:654] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.404227 2357 ts_tablet_manager.cc:1918] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.407739 2357 log.cc:1198] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/33f9b3e74ee2431a85a22fefaacfa40e
I20250110 02:51:32.408064 2357 ts_tablet_manager.cc:1939] T 33f9b3e74ee2431a85a22fefaacfa40e P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.409092 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 220) took 4263 ms. Trace:
I20250110 02:51:32.395778 1181 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.409209 2357 rpcz_store.cc:276] 0110 02:51:28.145507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.145817 (+ 310us) service_pool.cc:225] Handling call
0110 02:51:32.375598 (+4229781us) ts_tablet_manager.cc:1145] Deleting tablet 33f9b3e74ee2431a85a22fefaacfa40e
0110 02:51:32.375610 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.383136 (+ 7526us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.404144 (+ 21008us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.409075 (+ 4931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4229578}
I20250110 02:51:32.409377 1181 ts_tablet_manager.cc:1397] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250110 02:51:32.409596 2352 ts_tablet_manager.cc:1918] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.409968 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 33f9b3e74ee2431a85a22fefaacfa40e (table default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]) successfully deleted
I20250110 02:51:32.410305 2357 tablet_replica.cc:331] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.410235 1181 raft_consensus.cc:357] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.410435 2352 log.cc:1198] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/24c4042c38de4969889f4821176fc7bb
I20250110 02:51:32.410563 1181 raft_consensus.cc:383] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.410604 2905 consensus_queue.cc:1035] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.410682 1181 raft_consensus.cc:738] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.410799 2357 raft_consensus.cc:2238] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.410960 2357 raft_consensus.cc:2267] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.410914 1181 consensus_queue.cc:260] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.411430 2357 ts_tablet_manager.cc:1905] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.411840 2352 ts_tablet_manager.cc:1939] T 24c4042c38de4969889f4821176fc7bb P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.412424 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 309) took 4951 ms. Trace:
I20250110 02:51:32.412809 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 242) took 6089 ms. Trace:
I20250110 02:51:32.414546 2224 rpcz_store.cc:276] 0110 02:51:27.461017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.461261 (+ 244us) service_pool.cc:225] Handling call
0110 02:51:32.309274 (+4848013us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:32.309284 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.368968 (+ 59684us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.391288 (+ 22320us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.412400 (+ 21112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4847862}
I20250110 02:51:32.415309 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet c00e97e496b54dcf82b7182591d6f4aa (table default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]) successfully deleted
I20250110 02:51:32.413337 1181 ts_tablet_manager.cc:1428] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:32.415831 1181 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.417531 1181 tablet_bootstrap.cc:654] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.417276 2691 raft_consensus.cc:491] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.417816 2691 raft_consensus.cc:513] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.418498 2691 leader_election.cc:290] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:32.420082 853 leader_election.cc:343] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f9b379b357194d729108428cbaebd51a
I20250110 02:51:32.420342 2352 rpcz_store.cc:276] 0110 02:51:26.323118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.323158 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:32.384662 (+6061504us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:32.384670 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.398365 (+ 13695us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.409507 (+ 11142us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.412788 (+ 3281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6061373}
I20250110 02:51:32.421125 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 24c4042c38de4969889f4821176fc7bb (table default.Table_009562f3b1ed4d928d6fb21a604b051b [id=5245da3aab944e908b03ceff6a3e97ea]) successfully deleted
I20250110 02:51:32.421413 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 241) took 6098 ms. Trace:
I20250110 02:51:32.421645 2352 rpcz_store.cc:276] 0110 02:51:26.322951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.323181 (+ 230us) service_pool.cc:225] Handling call
0110 02:51:32.421185 (+6098004us) ts_tablet_manager.cc:1145] Deleting tablet b43bb790e85343308a1bb1e1110bfb9c
0110 02:51:32.421196 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.421398 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6097732}
W20250110 02:51:32.422214 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because the tablet was not found. No further retry: Not found: Tablet not found: b43bb790e85343308a1bb1e1110bfb9c
I20250110 02:51:32.422492 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 243) took 6098 ms. Trace:
I20250110 02:51:32.422745 2352 rpcz_store.cc:276] 0110 02:51:26.323599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.323633 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:32.422228 (+6098595us) ts_tablet_manager.cc:1145] Deleting tablet 019c0d60da5548bb86c93ccd89bf94ff
0110 02:51:32.422240 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.422471 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6098472}
W20250110 02:51:32.423379 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 019c0d60da5548bb86c93ccd89bf94ff because the tablet was not found. No further retry: Not found: Tablet not found: 019c0d60da5548bb86c93ccd89bf94ff
I20250110 02:51:32.423655 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 244) took 6099 ms. Trace:
I20250110 02:51:32.423934 2352 rpcz_store.cc:276] 0110 02:51:26.324165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.324213 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:32.423442 (+6099229us) ts_tablet_manager.cc:1145] Deleting tablet bcc9a4a842914a6f878d01194cd9a5f7
0110 02:51:32.423453 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.423636 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6099064}
W20250110 02:51:32.424409 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because the tablet was not found. No further retry: Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:32.424618 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 245) took 6100 ms. Trace:
I20250110 02:51:32.424854 2352 rpcz_store.cc:276] 0110 02:51:26.324549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.324590 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.424552 (+6099962us) ts_tablet_manager.cc:1145] Deleting tablet 1390f867a3a8436092d1920daff4c5b0
0110 02:51:32.424560 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.424605 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6099770}
W20250110 02:51:32.425328 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:32.425535 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 246) took 6100 ms. Trace:
I20250110 02:51:32.425704 2352 rpcz_store.cc:276] 0110 02:51:26.324888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.324929 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.425348 (+6100419us) ts_tablet_manager.cc:1145] Deleting tablet f6df074974974d81a83420e7a3a78070
0110 02:51:32.425355 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.425522 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6100289}
W20250110 02:51:32.426199 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f6df074974974d81a83420e7a3a78070 because the tablet was not found. No further retry: Not found: Tablet not found: f6df074974974d81a83420e7a3a78070
I20250110 02:51:32.426249 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 247) took 6101 ms. Trace:
I20250110 02:51:32.426580 2352 rpcz_store.cc:276] 0110 02:51:26.325265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.325307 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:32.426185 (+6100878us) ts_tablet_manager.cc:1145] Deleting tablet aa6057de9da04f2c82b8f25c5b4e2fe4
0110 02:51:32.426193 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.426237 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6100724}
W20250110 02:51:32.427122 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because the tablet was not found. No further retry: Not found: Tablet not found: aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:32.427366 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 248) took 6101 ms. Trace:
I20250110 02:51:32.427561 2352 rpcz_store.cc:276] 0110 02:51:26.325623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.325665 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:32.427168 (+6101503us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:32.427178 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.427350 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6101375}
W20250110 02:51:32.428126 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because the tablet was not found. No further retry: Not found: Tablet not found: eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:32.428469 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 249) took 6102 ms. Trace:
I20250110 02:51:32.428637 2352 rpcz_store.cc:276] 0110 02:51:26.325962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.326006 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:32.428140 (+6102134us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:32.428149 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.428198 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6101997}
W20250110 02:51:32.429107 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 27a7885a80414f2193400672aa16a911 because the tablet was not found. No further retry: Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
I20250110 02:51:32.429317 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 251) took 6102 ms. Trace:
I20250110 02:51:32.429474 2352 rpcz_store.cc:276] 0110 02:51:26.326827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.326884 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:32.429118 (+6102234us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:32.429126 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.429170 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6102081}
I20250110 02:51:32.417878 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 311) took 4955 ms. Trace:
I20250110 02:51:32.419411 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b379b357194d729108428cbaebd51a" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.429637 2224 rpcz_store.cc:276] 0110 02:51:27.461950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.461991 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.417756 (+4955765us) ts_tablet_manager.cc:1145] Deleting tablet 9e72ed2d31a8467f8c092b48d593c1cc
0110 02:51:32.417770 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.417858 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4955609}
I20250110 02:51:32.429971 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 312) took 4967 ms. Trace:
I20250110 02:51:32.430040 2224 rpcz_store.cc:276] 0110 02:51:27.462255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.462293 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:32.429883 (+4967590us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:32.429891 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.429956 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4967423}
W20250110 02:51:32.430181 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9e72ed2d31a8467f8c092b48d593c1cc because the tablet was not found. No further retry: Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:32.430392 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 314) took 4967 ms. Trace:
I20250110 02:51:32.430498 2224 rpcz_store.cc:276] 0110 02:51:27.462647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.462691 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:32.430295 (+4967604us) ts_tablet_manager.cc:1145] Deleting tablet 585320c0970c4097b195caaae33789b5
0110 02:51:32.430306 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.430360 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4967463}
W20250110 02:51:32.430603 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because the tablet was not found. No further retry: Not found: Tablet not found: 747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:32.430835 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 315) took 4968 ms. Trace:
I20250110 02:51:32.430910 2224 rpcz_store.cc:276] 0110 02:51:27.462734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.462776 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:32.430751 (+4967975us) ts_tablet_manager.cc:1145] Deleting tablet 9507a5283d8f4d07a293ba474c642298
0110 02:51:32.430761 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.430818 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4967846}
W20250110 02:51:32.430991 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 585320c0970c4097b195caaae33789b5 because the tablet was not found. No further retry: Not found: Tablet not found: 585320c0970c4097b195caaae33789b5
I20250110 02:51:32.431182 2224 tablet_replica.cc:331] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:32.431401 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because the tablet was not found. No further retry: Not found: Tablet not found: 9507a5283d8f4d07a293ba474c642298
W20250110 02:51:32.431906 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because the tablet was not found. No further retry: Not found: Tablet not found: b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:32.432186 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 252) took 6104 ms. Trace:
I20250110 02:51:32.433391 2352 rpcz_store.cc:276] 0110 02:51:26.327850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.327891 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.432116 (+6104225us) ts_tablet_manager.cc:1145] Deleting tablet d9018955f3884824beb2e7336987fc1c
0110 02:51:32.432125 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.432173 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6104079}
I20250110 02:51:32.433992 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 253) took 6105 ms. Trace:
I20250110 02:51:32.434268 2352 rpcz_store.cc:276] 0110 02:51:26.328209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.328299 (+ 90us) service_pool.cc:225] Handling call
0110 02:51:32.433928 (+6105629us) ts_tablet_manager.cc:1145] Deleting tablet 09f01daf4c67462ba75e1d0cf56cef12
0110 02:51:32.433937 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.433981 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6105483}
I20250110 02:51:32.434736 853 leader_election.cc:304] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.435932 2352 tablet_replica.cc:331] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.436146 2456 consensus_queue.cc:1035] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:32.437240 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d9018955f3884824beb2e7336987fc1c because the tablet was not found. No further retry: Not found: Tablet not found: d9018955f3884824beb2e7336987fc1c
W20250110 02:51:32.437479 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 09f01daf4c67462ba75e1d0cf56cef12 because the tablet was not found. No further retry: Not found: Tablet not found: 09f01daf4c67462ba75e1d0cf56cef12
I20250110 02:51:32.436009 2691 raft_consensus.cc:2743] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.439453 2352 raft_consensus.cc:2238] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.440061 2224 raft_consensus.cc:2238] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:32.440474 2224 raft_consensus.cc:2267] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.442167 2224 ts_tablet_manager.cc:1905] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.442291 2352 raft_consensus.cc:2267] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.443042 2352 ts_tablet_manager.cc:1905] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.445923 3105 raft_consensus.cc:491] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.446077 3105 raft_consensus.cc:513] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.446839 3105 leader_election.cc:290] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:32.442263 994 consensus_peers.cc:487] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.447404 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c2a19225ed477281b7ce60ae25b084" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.447773 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c2a19225ed477281b7ce60ae25b084" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:32.447991 994 consensus_peers.cc:487] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:32.448350 709 leader_election.cc:343] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:32.448582 711 leader_election.cc:343] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.448719 1188 tablet_bootstrap.cc:492] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.448688 711 leader_election.cc:304] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:32.448827 1188 ts_tablet_manager.cc:1397] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.001s
I20250110 02:51:32.448921 3105 raft_consensus.cc:2743] T b4c2a19225ed477281b7ce60ae25b084 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.449594 1188 raft_consensus.cc:357] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.449829 1188 raft_consensus.cc:383] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.449944 1188 raft_consensus.cc:738] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.450196 1188 consensus_queue.cc:260] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.450337 2357 ts_tablet_manager.cc:1918] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.450524 2357 log.cc:1198] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f69d0c201c524b7e95c9d3851b8dd8c2
I20250110 02:51:32.450023 1184 tablet_bootstrap.cc:492] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.451918 1184 ts_tablet_manager.cc:1397] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.005s
I20250110 02:51:32.452586 1184 raft_consensus.cc:357] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.452808 1184 raft_consensus.cc:383] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.452893 1184 raft_consensus.cc:738] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.453147 1184 consensus_queue.cc:260] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.453485 1188 ts_tablet_manager.cc:1428] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:32.453783 1188 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.453789 1184 ts_tablet_manager.cc:1428] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:32.454083 1184 tablet_bootstrap.cc:492] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.455302 1188 tablet_bootstrap.cc:654] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.457691 1184 tablet_bootstrap.cc:654] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.457816 2357 ts_tablet_manager.cc:1939] T f69d0c201c524b7e95c9d3851b8dd8c2 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.457008 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4baac522bef642cfaeb17dbb99a51306 (DEFAULT_TABLE table=default.Table_8cc1ce12b9c447f5a96de708a52c56ee [id=4e073a4acc514c9db104585a198423b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.459213 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4baac522bef642cfaeb17dbb99a51306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.463013 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 221) took 4316 ms. Trace:
W20250110 02:51:32.463739 853 consensus_peers.cc:487] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.465268 2357 rpcz_store.cc:276] 0110 02:51:28.146588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.146999 (+ 411us) service_pool.cc:225] Handling call
0110 02:51:32.410260 (+4263261us) ts_tablet_manager.cc:1145] Deleting tablet f69d0c201c524b7e95c9d3851b8dd8c2
0110 02:51:32.410270 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.416680 (+ 6410us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.450244 (+ 33564us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.462994 (+ 12750us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1920,"tablet-delete.queue_time_us":4263139}
I20250110 02:51:32.465750 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f69d0c201c524b7e95c9d3851b8dd8c2 (table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]) successfully deleted
I20250110 02:51:32.466151 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 222) took 4318 ms. Trace:
I20250110 02:51:32.466254 2357 rpcz_store.cc:276] 0110 02:51:28.147794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.147939 (+ 145us) service_pool.cc:225] Handling call
0110 02:51:32.466039 (+4318100us) ts_tablet_manager.cc:1145] Deleting tablet d6acae633eb14e5db87b1e36a069422c
0110 02:51:32.466049 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.466137 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4317970}
W20250110 02:51:32.466728 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d6acae633eb14e5db87b1e36a069422c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6acae633eb14e5db87b1e36a069422c already in progress: creating tablet
I20250110 02:51:32.467078 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 223) took 4318 ms. Trace:
I20250110 02:51:32.467262 2357 rpcz_store.cc:276] 0110 02:51:28.148940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.149220 (+ 280us) service_pool.cc:225] Handling call
0110 02:51:32.466983 (+4317763us) ts_tablet_manager.cc:1145] Deleting tablet e848586bec574a529caaf5e498db79cd
0110 02:51:32.466991 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.467064 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4317616}
I20250110 02:51:32.468187 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 4baac522bef642cfaeb17dbb99a51306 (DEFAULT_TABLE table=default.Table_8cc1ce12b9c447f5a96de708a52c56ee [id=4e073a4acc514c9db104585a198423b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.468546 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4baac522bef642cfaeb17dbb99a51306. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 02:51:32.468627 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e848586bec574a529caaf5e498db79cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e848586bec574a529caaf5e498db79cd already in progress: creating tablet
I20250110 02:51:32.468950 2357 tablet_replica.cc:331] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.469169 2357 raft_consensus.cc:2238] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.469345 2357 raft_consensus.cc:2267] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.470301 2357 ts_tablet_manager.cc:1905] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.470961 908 raft_consensus.cc:3054] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.473902 2549 consensus_queue.cc:1035] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.480125 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4baac522bef642cfaeb17dbb99a51306 (DEFAULT_TABLE table=default.Table_8cc1ce12b9c447f5a96de708a52c56ee [id=4e073a4acc514c9db104585a198423b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.480506 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4baac522bef642cfaeb17dbb99a51306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.480652 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 41e025e2898c4a28804a7ca2750cedff (DEFAULT_TABLE table=default.Table_df7558651391471ba93ced530307d670 [id=341c8fac1ccf453985c045c8481736d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.480904 756 tablet_service.cc:1467] Processing CreateTablet for tablet 41e025e2898c4a28804a7ca2750cedff (DEFAULT_TABLE table=default.Table_df7558651391471ba93ced530307d670 [id=341c8fac1ccf453985c045c8481736d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.480993 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e025e2898c4a28804a7ca2750cedff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.481222 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e025e2898c4a28804a7ca2750cedff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.482964 893 tablet_service.cc:1467] Processing CreateTablet for tablet 41e025e2898c4a28804a7ca2750cedff (DEFAULT_TABLE table=default.Table_df7558651391471ba93ced530307d670 [id=341c8fac1ccf453985c045c8481736d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.495114 896 tablet_service.cc:1467] Processing CreateTablet for tablet e1ff518696ff49529b1ec0f565e6a600 (DEFAULT_TABLE table=default.Table_2496754973df49aca9cc870a77615db3 [id=3186c662077e45fe85040faeaab72c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.496428 747 tablet_service.cc:1467] Processing CreateTablet for tablet e1ff518696ff49529b1ec0f565e6a600 (DEFAULT_TABLE table=default.Table_2496754973df49aca9cc870a77615db3 [id=3186c662077e45fe85040faeaab72c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.496778 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ff518696ff49529b1ec0f565e6a600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.496955 880 tablet_service.cc:1467] Processing CreateTablet for tablet ad85c2e893e24e7b8500e9047c985ad2 (DEFAULT_TABLE table=default.Table_74af5f41ea724f13baa721db4063a9d7 [id=65f17635bad9465e916dd396ccea91ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.497400 2352 ts_tablet_manager.cc:1918] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.497414 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad85c2e893e24e7b8500e9047c985ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.497536 2352 log.cc:1198] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7564208da99449cd97338a41b263e750
I20250110 02:51:32.497912 2352 ts_tablet_manager.cc:1939] T 7564208da99449cd97338a41b263e750 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.498013 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ff518696ff49529b1ec0f565e6a600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.499022 1181 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.499238 1181 ts_tablet_manager.cc:1397] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20250110 02:51:32.500295 1181 raft_consensus.cc:357] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.500479 1181 raft_consensus.cc:383] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.500531 1181 raft_consensus.cc:738] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.500739 1181 consensus_queue.cc:260] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.502566 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41e025e2898c4a28804a7ca2750cedff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.503289 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 254) took 6174 ms. Trace:
I20250110 02:51:32.503613 2352 rpcz_store.cc:276] 0110 02:51:26.328634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.328673 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:32.435880 (+6107207us) ts_tablet_manager.cc:1145] Deleting tablet 7564208da99449cd97338a41b263e750
0110 02:51:32.435888 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.459706 (+ 23818us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.497311 (+ 37605us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.503273 (+ 5962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6107100}
I20250110 02:51:32.504150 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 255) took 6175 ms. Trace:
I20250110 02:51:32.504576 2352 rpcz_store.cc:276] 0110 02:51:26.328968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.329006 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:32.504038 (+6175032us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:32.504046 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.504136 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6174906}
I20250110 02:51:32.504971 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 256) took 6175 ms. Trace:
I20250110 02:51:32.506531 1181 ts_tablet_manager.cc:1428] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:32.506587 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7564208da99449cd97338a41b263e750 (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]) successfully deleted
I20250110 02:51:32.506573 2352 rpcz_store.cc:276] 0110 02:51:26.329284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.329320 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.504893 (+6175573us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:32.504901 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.504959 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6175436}
I20250110 02:51:32.506875 1181 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
W20250110 02:51:32.506916 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b90cc15dbf47d4b4dbaf12d588abb0 already in progress: creating tablet
I20250110 02:51:32.507110 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 257) took 6177 ms. Trace:
W20250110 02:51:32.507150 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because the tablet was not found. No further retry: Not found: Tablet not found: 8f4778347f054b86af697dbe3b287263
I20250110 02:51:32.507197 2352 rpcz_store.cc:276] 0110 02:51:26.329709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.329802 (+ 93us) service_pool.cc:225] Handling call
0110 02:51:32.506895 (+6177093us) ts_tablet_manager.cc:1145] Deleting tablet 55edfa05075b4edcb70ef29e5ee413c6
0110 02:51:32.506917 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.507094 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6176924}
I20250110 02:51:32.508162 1184 tablet_bootstrap.cc:492] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.508307 1181 tablet_bootstrap.cc:654] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.508328 1184 ts_tablet_manager.cc:1397] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20250110 02:51:32.509483 1184 raft_consensus.cc:357] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.509660 1184 raft_consensus.cc:383] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.509712 1184 raft_consensus.cc:738] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.510615 1184 consensus_queue.cc:260] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.512382 1184 ts_tablet_manager.cc:1428] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:32.512776 2738 raft_consensus.cc:491] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.512972 2738 raft_consensus.cc:513] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.513512 2738 leader_election.cc:290] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.513691 1184 tablet_bootstrap.cc:492] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.514310 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.514506 772 raft_consensus.cc:2463] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.514025 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.514845 2352 tablet_replica.cc:331] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.514860 996 leader_election.cc:304] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:32.514900 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55edfa05075b4edcb70ef29e5ee413c6 already in progress: creating tablet
W20250110 02:51:32.515254 994 leader_election.cc:343] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.515558 2738 raft_consensus.cc:2798] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.515632 2738 raft_consensus.cc:491] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.515694 2738 raft_consensus.cc:3054] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.518344 1036 tablet_service.cc:1467] Processing CreateTablet for tablet ad85c2e893e24e7b8500e9047c985ad2 (DEFAULT_TABLE table=default.Table_74af5f41ea724f13baa721db4063a9d7 [id=65f17635bad9465e916dd396ccea91ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.518714 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad85c2e893e24e7b8500e9047c985ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.518219 2738 raft_consensus.cc:513] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.520181 1184 tablet_bootstrap.cc:654] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.520411 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.520808 2352 raft_consensus.cc:2238] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:32.520923 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c0aaf2050f481daa8cdac4761fe705" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:32.521011 994 leader_election.cc:343] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.521076 772 raft_consensus.cc:3054] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.521129 2738 leader_election.cc:290] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.521169 2352 raft_consensus.cc:2267] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.523699 772 raft_consensus.cc:2463] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.524060 996 leader_election.cc:304] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.524142 2357 ts_tablet_manager.cc:1918] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.524223 2357 log.cc:1198] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:32.524582 2357 ts_tablet_manager.cc:1939] T 4de67338a26f4febb5dd7611a72b4723 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.524747 2456 raft_consensus.cc:2798] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.524848 2456 raft_consensus.cc:695] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.525074 2456 consensus_queue.cc:237] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.521646 2352 ts_tablet_manager.cc:1905] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.526329 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2312c1ee4a134caaa68980a224dd283e (DEFAULT_TABLE table=default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.526731 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2312c1ee4a134caaa68980a224dd283e (DEFAULT_TABLE table=default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.526979 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2312c1ee4a134caaa68980a224dd283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.527042 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2312c1ee4a134caaa68980a224dd283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.531284 1188 tablet_bootstrap.cc:492] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.531464 1188 ts_tablet_manager.cc:1397] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20250110 02:51:32.532140 1188 raft_consensus.cc:357] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.532375 1188 raft_consensus.cc:383] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.532467 1188 raft_consensus.cc:738] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.532737 1188 consensus_queue.cc:260] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.533325 1188 ts_tablet_manager.cc:1428] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.533805 1188 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.533669 1030 tablet_service.cc:1467] Processing CreateTablet for tablet e1ff518696ff49529b1ec0f565e6a600 (DEFAULT_TABLE table=default.Table_2496754973df49aca9cc870a77615db3 [id=3186c662077e45fe85040faeaab72c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.534547 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1ff518696ff49529b1ec0f565e6a600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.536067 1188 tablet_bootstrap.cc:654] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.525310 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 224) took 4304 ms. Trace:
I20250110 02:51:32.526312 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2312c1ee4a134caaa68980a224dd283e (DEFAULT_TABLE table=default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.543679 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2312c1ee4a134caaa68980a224dd283e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.548763 2352 ts_tablet_manager.cc:1918] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.548869 2352 log.cc:1198] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/452223af5f6a4836a0f582d394a9453e
I20250110 02:51:32.549230 2352 ts_tablet_manager.cc:1939] T 452223af5f6a4836a0f582d394a9453e P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.549921 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 258) took 6219 ms. Trace:
I20250110 02:51:32.550009 2352 rpcz_store.cc:276] 0110 02:51:26.330164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.330274 (+ 110us) service_pool.cc:225] Handling call
0110 02:51:32.514804 (+6184530us) ts_tablet_manager.cc:1145] Deleting tablet 452223af5f6a4836a0f582d394a9453e
0110 02:51:32.514813 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.530540 (+ 15727us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.548684 (+ 18144us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.549893 (+ 1209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6184420}
I20250110 02:51:32.550403 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 452223af5f6a4836a0f582d394a9453e (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]) successfully deleted
I20250110 02:51:32.538267 756 tablet_service.cc:1467] Processing CreateTablet for tablet ad85c2e893e24e7b8500e9047c985ad2 (DEFAULT_TABLE table=default.Table_74af5f41ea724f13baa721db4063a9d7 [id=65f17635bad9465e916dd396ccea91ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 02:51:32.556972 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad85c2e893e24e7b8500e9047c985ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 02:51:32.543251 2357 rpcz_store.cc:276] 0110 02:51:28.221136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.221182 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:32.468884 (+4247702us) ts_tablet_manager.cc:1145] Deleting tablet 4de67338a26f4febb5dd7611a72b4723
0110 02:51:32.468896 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.505547 (+ 36651us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.524058 (+ 18511us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.525293 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193427}
I20250110 02:51:32.559672 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 4de67338a26f4febb5dd7611a72b4723 (table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]) successfully deleted
I20250110 02:51:32.560019 2357 tablet_replica.cc:331] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.560237 2357 raft_consensus.cc:2238] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.560428 2357 raft_consensus.cc:2267] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.560993 2357 ts_tablet_manager.cc:1905] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.561053 1188 tablet_bootstrap.cc:492] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.561190 1188 ts_tablet_manager.cc:1397] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20250110 02:51:32.561841 1188 raft_consensus.cc:357] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.562031 1188 raft_consensus.cc:383] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.562090 1188 raft_consensus.cc:738] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.562289 1188 consensus_queue.cc:260] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.562804 1188 ts_tablet_manager.cc:1428] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:32.563051 1188 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.564529 1188 tablet_bootstrap.cc:654] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.565315 1184 tablet_bootstrap.cc:492] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.565459 1184 ts_tablet_manager.cc:1397] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.003s
I20250110 02:51:32.566012 1181 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.566164 1181 ts_tablet_manager.cc:1397] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20250110 02:51:32.566243 1184 raft_consensus.cc:357] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.566448 1184 raft_consensus.cc:383] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.566519 1184 raft_consensus.cc:738] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.566839 1181 raft_consensus.cc:357] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.567028 1181 raft_consensus.cc:383] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.567091 1181 raft_consensus.cc:738] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.567075 1184 consensus_queue.cc:260] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.567317 1181 consensus_queue.cc:260] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.567772 1181 ts_tablet_manager.cc:1428] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:32.567701 1184 ts_tablet_manager.cc:1428] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.568032 1181 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.568037 1184 tablet_bootstrap.cc:492] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.570009 3105 raft_consensus.cc:491] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.570119 3105 raft_consensus.cc:513] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.570669 3105 leader_election.cc:290] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:32.571768 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07acc4681ae94a069003ed75fcce5141" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:32.571859 996 consensus_peers.cc:487] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f69d0c201c524b7e95c9d3851b8dd8c2. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.572074 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07acc4681ae94a069003ed75fcce5141" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:32.572320 711 leader_election.cc:343] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:32.572531 709 leader_election.cc:343] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.572633 709 leader_election.cc:304] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:32.572993 3105 raft_consensus.cc:2743] T 07acc4681ae94a069003ed75fcce5141 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.575950 1181 tablet_bootstrap.cc:654] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.579957 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 259) took 6249 ms. Trace:
I20250110 02:51:32.580051 2352 rpcz_store.cc:276] 0110 02:51:26.330242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.330465 (+ 223us) service_pool.cc:225] Handling call
0110 02:51:32.579838 (+6249373us) ts_tablet_manager.cc:1145] Deleting tablet 0127de45264f4660b843734a530eb310
0110 02:51:32.579848 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.579937 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6249257}
I20250110 02:51:32.580150 2224 ts_tablet_manager.cc:1918] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:32.580475 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 0127de45264f4660b843734a530eb310 because the tablet was not found. No further retry: Not found: Tablet not found: 0127de45264f4660b843734a530eb310
I20250110 02:51:32.581959 2224 log.cc:1198] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/08705d260c0a462b94ba6b0dc5121123
I20250110 02:51:32.583226 2224 ts_tablet_manager.cc:1939] T 08705d260c0a462b94ba6b0dc5121123 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.585125 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 316) took 5122 ms. Trace:
I20250110 02:51:32.587539 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 260) took 6256 ms. Trace:
I20250110 02:51:32.587635 2352 rpcz_store.cc:276] 0110 02:51:26.330714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.330748 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:32.587397 (+6256649us) ts_tablet_manager.cc:1145] Deleting tablet 7e554d00d05b4fa7a14c9b6159f9d726
0110 02:51:32.587410 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.587520 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6256517}
W20250110 02:51:32.588056 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7e554d00d05b4fa7a14c9b6159f9d726 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e554d00d05b4fa7a14c9b6159f9d726 already in progress: creating tablet
I20250110 02:51:32.588308 2224 rpcz_store.cc:276] 0110 02:51:27.462803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.462861 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:32.431133 (+4968272us) ts_tablet_manager.cc:1145] Deleting tablet 08705d260c0a462b94ba6b0dc5121123
0110 02:51:32.431150 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.457602 (+ 26452us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.577844 (+120242us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.585104 (+ 7260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4968161}
I20250110 02:51:32.588655 2352 tablet_replica.cc:331] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:32.588805 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 08705d260c0a462b94ba6b0dc5121123 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:32.588878 2352 raft_consensus.cc:2238] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.589047 2352 raft_consensus.cc:2267] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.589691 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 318) took 5126 ms. Trace:
I20250110 02:51:32.589813 2224 rpcz_store.cc:276] 0110 02:51:27.462957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.462988 (+ 31us) service_pool.cc:225] Handling call
0110 02:51:32.589602 (+5126614us) ts_tablet_manager.cc:1145] Deleting tablet 6e4d3270e3a04cc99f90943b9573ec7e
0110 02:51:32.589613 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.589675 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5126467}
W20250110 02:51:32.590303 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6e4d3270e3a04cc99f90943b9573ec7e because the tablet was not found. No further retry: Not found: Tablet not found: 6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:32.590665 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 319) took 5127 ms. Trace:
I20250110 02:51:32.592255 2224 rpcz_store.cc:276] 0110 02:51:27.463027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.463057 (+ 30us) service_pool.cc:225] Handling call
0110 02:51:32.590584 (+5127527us) ts_tablet_manager.cc:1145] Deleting tablet 32dd1e726f1a41d0894737b3ef3caf8f
0110 02:51:32.590593 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.590650 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5127416}
W20250110 02:51:32.592759 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 32dd1e726f1a41d0894737b3ef3caf8f because the tablet was not found. No further retry: Not found: Tablet not found: 32dd1e726f1a41d0894737b3ef3caf8f
I20250110 02:51:32.593106 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 320) took 5130 ms. Trace:
I20250110 02:51:32.593080 2456 raft_consensus.cc:491] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.593300 2456 raft_consensus.cc:513] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.594780 2352 ts_tablet_manager.cc:1905] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.594735 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:32.595825 994 leader_election.cc:343] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.596490 2456 leader_election.cc:290] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.593222 2224 rpcz_store.cc:276] 0110 02:51:27.463126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.463154 (+ 28us) service_pool.cc:225] Handling call
0110 02:51:32.593025 (+5129871us) ts_tablet_manager.cc:1145] Deleting tablet 95b14d7592024a34baf7756d92e546d9
0110 02:51:32.593034 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.593092 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5129735}
W20250110 02:51:32.597389 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 95b14d7592024a34baf7756d92e546d9 because the tablet was not found. No further retry: Not found: Tablet not found: 95b14d7592024a34baf7756d92e546d9
I20250110 02:51:32.598450 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.598639 771 raft_consensus.cc:2463] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.599203 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 317) took 5136 ms. Trace:
I20250110 02:51:32.599282 2224 rpcz_store.cc:276] 0110 02:51:27.462885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.463011 (+ 126us) service_pool.cc:225] Handling call
0110 02:51:32.599123 (+5136112us) ts_tablet_manager.cc:1145] Deleting tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
0110 02:51:32.599132 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.599189 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5135812}
W20250110 02:51:32.599704 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cdb7d0b9df3d48f0bb10eb6bc9b2cd6b because the tablet was not found. No further retry: Not found: Tablet not found: cdb7d0b9df3d48f0bb10eb6bc9b2cd6b
I20250110 02:51:32.601258 996 leader_election.cc:304] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.601341 2357 ts_tablet_manager.cc:1918] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.601920 2898 raft_consensus.cc:2798] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.602008 2898 raft_consensus.cc:491] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.602079 2898 raft_consensus.cc:3054] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.602654 1181 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.602787 1181 ts_tablet_manager.cc:1397] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20250110 02:51:32.603447 1181 raft_consensus.cc:357] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.603605 1181 raft_consensus.cc:383] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.603646 1181 raft_consensus.cc:738] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.603837 1181 consensus_queue.cc:260] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.604071 2906 consensus_queue.cc:1035] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.604394 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:32.604775 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:32.604984 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:32.604954 2898 raft_consensus.cc:513] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.605137 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:32.605512 2898 leader_election.cc:290] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.605782 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.605858 1181 ts_tablet_manager.cc:1428] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:32.605849 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
W20250110 02:51:32.606446 994 leader_election.cc:343] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.606925 1181 tablet_bootstrap.cc:492] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.606577 1444 catalog_manager.cc:5526] T ae3033595f824affb2d300c3964e1e8e P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.607348 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:32.607736 1444 catalog_manager.cc:5526] T e27f7ffc50494854ba34e1fbce9bee1b P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.608867 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:32.609180 2357 log.cc:1198] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/7c2c7a313ba94e5ea85e45276658ba04
I20250110 02:51:32.609534 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.609644 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:32.609714 774 raft_consensus.cc:3054] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.610728 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:32.611270 1184 tablet_bootstrap.cc:654] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.611663 2357 ts_tablet_manager.cc:1939] T 7c2c7a313ba94e5ea85e45276658ba04 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.611685 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3aee71c51a96482fbb527576867bbaa9 (table default.Table_a99fe640359046aa8ff288db7a687676 [id=c9a367ed70c742a18d7dc76e72b37300]). Expected version 1 got 0
I20250110 02:51:32.612377 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:32.612560 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 225) took 4333 ms. Trace:
I20250110 02:51:32.613247 1444 catalog_manager.cc:5526] T 069d19975e2a4bb0a23b4e9983f411d3 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:32.614172 1444 catalog_manager.cc:5526] T eadb829d657346eabcb7d2aa34b33fe2 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.615221 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 9e28d8769f364e10aad54b60fb221f35 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:32.615281 2352 ts_tablet_manager.cc:1918] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.615420 2352 log.cc:1198] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/08705d260c0a462b94ba6b0dc5121123
I20250110 02:51:32.615581 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:32.615622 2224 tablet_replica.cc:331] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.615789 2224 raft_consensus.cc:2238] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.615832 2352 ts_tablet_manager.cc:1939] T 08705d260c0a462b94ba6b0dc5121123 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:32.616796 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:32.617458 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:32.617619 1188 tablet_bootstrap.cc:492] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.617779 1188 ts_tablet_manager.cc:1397] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.001s
I20250110 02:51:32.618147 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:32.619022 774 raft_consensus.cc:2463] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.619351 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:32.612665 2357 rpcz_store.cc:276] 0110 02:51:28.278822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.278890 (+ 68us) service_pool.cc:225] Handling call
0110 02:51:32.559972 (+4281082us) ts_tablet_manager.cc:1145] Deleting tablet 7c2c7a313ba94e5ea85e45276658ba04
0110 02:51:32.559982 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.564447 (+ 4465us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.601249 (+ 36802us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.612542 (+ 11293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4280912}
I20250110 02:51:32.619966 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 7c2c7a313ba94e5ea85e45276658ba04 (table default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]) successfully deleted
I20250110 02:51:32.620013 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 226) took 4256 ms. Trace:
I20250110 02:51:32.620081 2357 rpcz_store.cc:276] 0110 02:51:28.363935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.363987 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:32.619897 (+4255910us) ts_tablet_manager.cc:1145] Deleting tablet 9b937188bf2949af8f54bade4e0ad984
0110 02:51:32.619918 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.620002 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255693}
I20250110 02:51:32.620065 1444 catalog_manager.cc:5526] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
W20250110 02:51:32.620630 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9b937188bf2949af8f54bade4e0ad984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b937188bf2949af8f54bade4e0ad984 already in progress: creating tablet
I20250110 02:51:32.619623 996 leader_election.cc:304] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.621342 2898 raft_consensus.cc:2798] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.621477 2898 raft_consensus.cc:695] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.621928 2898 consensus_queue.cc:237] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.620790 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 227) took 4256 ms. Trace:
I20250110 02:51:32.625746 2738 consensus_queue.cc:1035] T 13683fbe9d654575b28108de8b77f943 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:32.625089 2357 rpcz_store.cc:276] 0110 02:51:28.364352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.364550 (+ 198us) service_pool.cc:225] Handling call
0110 02:51:32.620676 (+4256126us) ts_tablet_manager.cc:1145] Deleting tablet e3fc9826ef554a5699e365d10d02e97a
0110 02:51:32.620685 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.620771 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4255983}
W20250110 02:51:32.626596 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e3fc9826ef554a5699e365d10d02e97a because the tablet was not found. No further retry: Not found: Tablet not found: e3fc9826ef554a5699e365d10d02e97a
I20250110 02:51:32.626802 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 228) took 4262 ms. Trace:
I20250110 02:51:32.630676 1181 tablet_bootstrap.cc:654] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.631353 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
W20250110 02:51:32.631567 853 consensus_peers.cc:487] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c2c7a313ba94e5ea85e45276658ba04. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.632218 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:32.632678 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:32.626878 2357 rpcz_store.cc:276] 0110 02:51:28.364447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.364495 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:32.626731 (+4262236us) ts_tablet_manager.cc:1145] Deleting tablet 19b90cc15dbf47d4b4dbaf12d588abb0
0110 02:51:32.626739 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.626789 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4235885}
I20250110 02:51:32.633384 2357 tablet_replica.cc:331] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.633621 2357 raft_consensus.cc:2238] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:32.633843 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 because the tablet was not found. No further retry: Not found: Tablet not found: 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:32.634160 2357 raft_consensus.cc:2267] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.634925 2357 ts_tablet_manager.cc:1905] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.638933 1188 raft_consensus.cc:357] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.639228 1188 raft_consensus.cc:383] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.639329 1188 raft_consensus.cc:738] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.639566 1188 consensus_queue.cc:260] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.640614 1188 ts_tablet_manager.cc:1428] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250110 02:51:32.641938 2456 raft_consensus.cc:491] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.642099 2456 raft_consensus.cc:513] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.643716 2456 leader_election.cc:290] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.643770 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.643945 771 raft_consensus.cc:2463] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.644316 996 leader_election.cc:304] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.644610 2456 raft_consensus.cc:2798] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.644706 2456 raft_consensus.cc:491] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.644806 2456 raft_consensus.cc:3054] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.645078 2738 raft_consensus.cc:491] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.645221 2738 raft_consensus.cc:513] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.646256 2738 leader_election.cc:290] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.646795 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.646973 771 raft_consensus.cc:2463] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.633844 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4f1d7940a1574af1804d5f5139ca7494 (table default.Table_b4223b31a5974abb9725bceb8026a8ae [id=4e2fb95ceb8c43fc920b1cd9e73557df]). Expected version 1 got 0
I20250110 02:51:32.647377 996 leader_election.cc:304] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.647377 1444 catalog_manager.cc:5526] T 718d3add9d0a4e4b8633f2c977e316e6 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } }
I20250110 02:51:32.647614 2738 raft_consensus.cc:2798] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.647722 2738 raft_consensus.cc:491] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.647809 2738 raft_consensus.cc:3054] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.647970 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:32.648679 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:32.648885 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:32.649291 1444 catalog_manager.cc:5526] T e2ab2376576448deb87f607b3e5a4f1c P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.649767 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:32.649947 1444 catalog_manager.cc:5526] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.651453 2738 raft_consensus.cc:513] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.654420 2224 raft_consensus.cc:2267] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.657395 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.657579 774 raft_consensus.cc:3054] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.657881 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.657913 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.657877 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:32.658596 994 leader_election.cc:343] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:32.658777 994 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:32.659036 994 leader_election.cc:343] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.659776 2738 leader_election.cc:290] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.660349 774 raft_consensus.cc:2463] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.660730 996 leader_election.cc:304] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.661067 2738 raft_consensus.cc:2798] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.661549 1188 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.661912 3150 tablet.cc:1721] T bfd120d8edf14151b7ebed3b74e2b399 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:32.662840 2738 raft_consensus.cc:695] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.663280 2738 consensus_queue.cc:237] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.663674 1188 tablet_bootstrap.cc:654] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.616729 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 261) took 6285 ms. Trace:
I20250110 02:51:32.671432 2352 rpcz_store.cc:276] 0110 02:51:26.331035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.331071 (+ 36us) service_pool.cc:225] Handling call
0110 02:51:32.588606 (+6257535us) ts_tablet_manager.cc:1145] Deleting tablet 08705d260c0a462b94ba6b0dc5121123
0110 02:51:32.588618 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.604166 (+ 15548us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.615162 (+ 10996us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.616703 (+ 1541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6257388}
I20250110 02:51:32.671986 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 08705d260c0a462b94ba6b0dc5121123 (table default.Table_ee284925f5344933a090d3f345f45b53 [id=5d030d88106f4a8bb3e7fa992ac5b8c9]) successfully deleted
I20250110 02:51:32.673197 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 262) took 6332 ms. Trace:
I20250110 02:51:32.673305 2352 rpcz_store.cc:276] 0110 02:51:26.341112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.341156 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:32.673062 (+6331906us) ts_tablet_manager.cc:1145] Deleting tablet dad0d561a8ef498ba13bc813d4cd82fa
0110 02:51:32.673076 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.673177 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6331749}
W20250110 02:51:32.673676 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet dad0d561a8ef498ba13bc813d4cd82fa because the tablet was not found. No further retry: Not found: Tablet not found: dad0d561a8ef498ba13bc813d4cd82fa
I20250110 02:51:32.663879 2224 ts_tablet_manager.cc:1905] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.664271 1181 tablet_bootstrap.cc:492] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.679126 1181 ts_tablet_manager.cc:1397] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.002s
I20250110 02:51:32.679137 1444 catalog_manager.cc:5526] T 71ffa250a3554a7b8f3a9c34ee7e4674 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.679638 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_614d56e4a7544c27aa6f21776b2138f3 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 3 got 0
I20250110 02:51:32.679878 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:32.680114 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:32.679939 1181 raft_consensus.cc:357] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.680289 1181 raft_consensus.cc:383] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.680384 1181 raft_consensus.cc:738] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.680527 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:32.680785 1181 consensus_queue.cc:260] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.680979 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:32.681315 1181 ts_tablet_manager.cc:1428] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:32.681620 1181 tablet_bootstrap.cc:492] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.681980 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:32.682792 2357 ts_tablet_manager.cc:1918] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.682916 2357 log.cc:1198] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/213939cf13354ce380fb7cda5b38676f
I20250110 02:51:32.683240 2357 ts_tablet_manager.cc:1939] T 213939cf13354ce380fb7cda5b38676f P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.684131 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 229) took 4261 ms. Trace:
I20250110 02:51:32.684211 2357 rpcz_store.cc:276] 0110 02:51:28.422872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.423206 (+ 334us) service_pool.cc:225] Handling call
0110 02:51:32.633337 (+4210131us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:32.633346 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.665331 (+ 31985us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.682105 (+ 16774us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.684113 (+ 2008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4209951}
I20250110 02:51:32.684635 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 213939cf13354ce380fb7cda5b38676f (table default.Table_0e2df09ed397432f8697c6baf59aaa53 [id=620801551d814277ae80ea40ecb3532b]) successfully deleted
I20250110 02:51:32.685757 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:32.685847 1184 tablet_bootstrap.cc:492] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.686395 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:32.686656 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 263) took 6345 ms. Trace:
I20250110 02:51:32.686756 2352 rpcz_store.cc:276] 0110 02:51:26.341484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.341525 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.686543 (+6345018us) ts_tablet_manager.cc:1145] Deleting tablet d10ace37b96844328ccdb821dad3c5ea
0110 02:51:32.686555 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.686635 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6344860}
I20250110 02:51:32.689776 1184 ts_tablet_manager.cc:1397] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.008s
I20250110 02:51:32.691193 1181 tablet_bootstrap.cc:654] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.693725 3149 tablet.cc:1721] T bfd120d8edf14151b7ebed3b74e2b399 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 02:51:32.687152 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d10ace37b96844328ccdb821dad3c5ea because the tablet was not found. No further retry: Not found: Tablet not found: d10ace37b96844328ccdb821dad3c5ea
I20250110 02:51:32.695226 1444 catalog_manager.cc:5526] T 04a4e7142e83493b98f71487df3f304d P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.695636 1184 raft_consensus.cc:357] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.696074 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 264) took 6354 ms. Trace:
I20250110 02:51:32.697136 1444 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:32.697710 2352 rpcz_store.cc:276] 0110 02:51:26.341883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.341923 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:32.695963 (+6354040us) ts_tablet_manager.cc:1145] Deleting tablet f66603a0ee8c4082807c5a6aabe7e339
0110 02:51:32.695974 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.696053 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6353911}
W20250110 02:51:32.698169 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f66603a0ee8c4082807c5a6aabe7e339 because the tablet was not found. No further retry: Not found: Tablet not found: f66603a0ee8c4082807c5a6aabe7e339
I20250110 02:51:32.698863 1456 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet fe86be26d1b544c08b231f20085d3159
I20250110 02:51:32.699404 1456 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 178) took 2083 ms. Trace:
I20250110 02:51:32.699509 1456 rpcz_store.cc:276] 0110 02:51:30.615921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.812699 (+1196778us) service_pool.cc:225] Handling call
0110 02:51:31.813053 (+ 354us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:31.813182 (+ 129us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:31.813235 (+ 53us) catalog_manager.cc:2762] Locking tablets
0110 02:51:31.870917 (+ 57682us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:31.881380 (+ 10463us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:31.881450 (+ 70us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:32.698800 (+817350us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:32.699368 (+ 568us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.871792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.871903 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.871911 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.871914 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:31.872026 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.872040 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.872047 (+ 7us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.872160 (+ 113us) tablet.cc:698] Row locks acquired
0110 02:51:31.872164 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.872205 (+ 41us) write_op.cc:270] Start()
0110 02:51:31.872244 (+ 39us) write_op.cc:276] Timestamp: P: 1736477491872147 usec, L: 0
0110 02:51:31.872262 (+ 18us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.872473 (+ 211us) log.cc:844] Serialized 765 byte log entry
0110 02:51:31.879108 (+ 6635us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.879228 (+ 120us) write_op.cc:301] APPLY: starting
0110 02:51:31.879244 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.879288 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.879311 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.879371 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.879402 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:31.879417 (+ 15us) write_op.cc:312] APPLY: finished
0110 02:51:31.879498 (+ 81us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.879553 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.879572 (+ 19us) write_op.cc:454] Released schema lock
0110 02:51:31.879578 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.879639 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.879642 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":403,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":777,"raft.queue_time_us":426,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":201,"replication_time_us":6821}]]}
I20250110 02:51:32.665084 2456 raft_consensus.cc:513] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.700980 2456 leader_election.cc:290] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.702020 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.702553 2352 tablet_replica.cc:331] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:32.703294 994 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:32.703502 994 consensus_peers.cc:487] T 9395aa9b39d04ac9b6cecf106bc91499 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.704216 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:32.704327 2352 raft_consensus.cc:2238] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.704712 893 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:32.705772 2352 raft_consensus.cc:2267] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.706305 2352 ts_tablet_manager.cc:1905] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.708842 1181 tablet_bootstrap.cc:492] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.709014 1181 ts_tablet_manager.cc:1397] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250110 02:51:32.709000 1184 raft_consensus.cc:383] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.703061 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed058931194480ca1a27130bfa8ee32" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.709128 1448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet e55ee565b7f14936a2cb4cbed716c153
I20250110 02:51:32.709260 772 raft_consensus.cc:3054] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.709607 1448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 160) took 3913 ms. Trace:
I20250110 02:51:32.709695 1448 rpcz_store.cc:276] 0110 02:51:28.795968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.201011 (+2405043us) service_pool.cc:225] Handling call
0110 02:51:31.201424 (+ 413us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:31.201557 (+ 133us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:31.201613 (+ 56us) catalog_manager.cc:2762] Locking tablets
0110 02:51:31.324569 (+122956us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:31.331102 (+ 6533us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:31.331205 (+ 103us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:32.709074 (+1377869us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:32.709554 (+ 480us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.325428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.325548 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.325556 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.325560 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:31.325716 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.325733 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.325741 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:31.325859 (+ 118us) tablet.cc:698] Row locks acquired
0110 02:51:31.325863 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.325900 (+ 37us) write_op.cc:270] Start()
0110 02:51:31.325934 (+ 34us) write_op.cc:276] Timestamp: P: 1736477491325845 usec, L: 0
0110 02:51:31.325940 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.326171 (+ 231us) log.cc:844] Serialized 725 byte log entry
0110 02:51:31.329398 (+ 3227us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.330575 (+ 1177us) write_op.cc:301] APPLY: starting
0110 02:51:31.330591 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.330633 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.330685 (+ 52us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.330749 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.330785 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:31.330794 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:31.330862 (+ 68us) log.cc:844] Serialized 29 byte log entry
0110 02:51:31.330912 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.330932 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:31.330939 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.331006 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.331011 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1124,"apply.run_cpu_time_us":423,"apply.run_wall_time_us":424,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":845,"prepare.run_wall_time_us":845,"raft.queue_time_us":66,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":272,"replication_time_us":3425}]]}
I20250110 02:51:32.713112 772 raft_consensus.cc:2463] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.709077 1184 raft_consensus.cc:738] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.704063 756 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:32.715538 1184 consensus_queue.cc:260] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.715998 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e55ee565b7f14936a2cb4cbed716c153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:32.718622 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e55ee565b7f14936a2cb4cbed716c153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:32.724820 1184 ts_tablet_manager.cc:1428] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250110 02:51:32.732081 1184 tablet_bootstrap.cc:492] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.713574 996 leader_election.cc:304] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.716398 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e55ee565b7f14936a2cb4cbed716c153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:32.738814 2456 raft_consensus.cc:2798] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.739025 2456 raft_consensus.cc:695] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.739321 2456 consensus_queue.cc:237] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.741114 1188 tablet_bootstrap.cc:492] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.742043 1188 ts_tablet_manager.cc:1397] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.004s
I20250110 02:51:32.739641 1181 raft_consensus.cc:357] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.743813 1181 raft_consensus.cc:383] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.744091 1188 raft_consensus.cc:357] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.744289 1181 raft_consensus.cc:738] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:32.744525 1430 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_614d56e4a7544c27aa6f21776b2138f3 [id=0a7a61acdae94ee392948eaa9495dfcb], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24cf81039d5e4a47b426ea5d2f21acc2 consensus state has no leader
I20250110 02:51:32.746234 1181 consensus_queue.cc:260] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.748883 1181 ts_tablet_manager.cc:1428] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250110 02:51:32.750447 1181 tablet_bootstrap.cc:492] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.689616 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 230) took 4266 ms. Trace:
I20250110 02:51:32.756827 1188 raft_consensus.cc:383] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.749924 3158 tablet.cc:1721] T bfd120d8edf14151b7ebed3b74e2b399 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:32.758795 1188 raft_consensus.cc:738] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.767264 1188 consensus_queue.cc:260] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.767915 1188 ts_tablet_manager.cc:1428] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250110 02:51:32.769454 1181 tablet_bootstrap.cc:654] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.750104 1184 tablet_bootstrap.cc:654] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.753841 2357 rpcz_store.cc:276] 0110 02:51:28.423332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.423879 (+ 547us) service_pool.cc:225] Handling call
0110 02:51:32.689522 (+4265643us) ts_tablet_manager.cc:1145] Deleting tablet 8f4778347f054b86af697dbe3b287263
0110 02:51:32.689533 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.689599 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4265502}
I20250110 02:51:32.770836 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 236) took 4347 ms. Trace:
W20250110 02:51:32.770936 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8f4778347f054b86af697dbe3b287263 because the tablet was not found. No further retry: Not found: Tablet not found: 8f4778347f054b86af697dbe3b287263
I20250110 02:51:32.770991 2357 rpcz_store.cc:276] 0110 02:51:28.423837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.424045 (+ 208us) service_pool.cc:225] Handling call
0110 02:51:32.770729 (+4346684us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:32.770742 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.770818 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4346539}
I20250110 02:51:32.771291 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 232) took 4347 ms. Trace:
W20250110 02:51:32.771445 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because the tablet was not found. No further retry: Not found: Tablet not found: b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:32.771365 2357 rpcz_store.cc:276] 0110 02:51:28.423514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.425951 (+ 2437us) service_pool.cc:225] Handling call
0110 02:51:32.771208 (+4345257us) ts_tablet_manager.cc:1145] Deleting tablet 6e4d3270e3a04cc99f90943b9573ec7e
0110 02:51:32.771218 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.771275 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4342899}
I20250110 02:51:32.771941 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 234) took 4348 ms. Trace:
W20250110 02:51:32.772027 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6e4d3270e3a04cc99f90943b9573ec7e because the tablet was not found. No further retry: Not found: Tablet not found: 6e4d3270e3a04cc99f90943b9573ec7e
I20250110 02:51:32.772012 2357 rpcz_store.cc:276] 0110 02:51:28.423686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.425998 (+ 2312us) service_pool.cc:225] Handling call
0110 02:51:32.771856 (+4345858us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:32.771866 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.771926 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4343421}
I20250110 02:51:32.772315 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 233) took 4348 ms. Trace:
I20250110 02:51:32.772384 2357 rpcz_store.cc:276] 0110 02:51:28.423613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.425977 (+ 2364us) service_pool.cc:225] Handling call
0110 02:51:32.772217 (+4346240us) ts_tablet_manager.cc:1145] Deleting tablet 4aa78a9ef5eb443da0c8983a7190440b
0110 02:51:32.772227 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.772302 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4343672}
W20250110 02:51:32.772472 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because the tablet was not found. No further retry: Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:32.772800 2908 raft_consensus.cc:491] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:32.772894 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:32.772928 2908 raft_consensus.cc:513] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.773530 2908 leader_election.cc:290] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.774662 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.766885 2224 ts_tablet_manager.cc:1918] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.774113 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25bbf8aa71e478587093627c715310f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.776437 2224 log.cc:1198] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f63d69d04b494a848653636c9e259105
I20250110 02:51:32.776525 853 leader_election.cc:304] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.776760 2908 raft_consensus.cc:2743] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.776965 2224 ts_tablet_manager.cc:1939] T f63d69d04b494a848653636c9e259105 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.777717 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 321) took 5314 ms. Trace:
I20250110 02:51:32.778479 2224 rpcz_store.cc:276] 0110 02:51:27.463204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.463331 (+ 127us) service_pool.cc:225] Handling call
0110 02:51:32.615573 (+5152242us) ts_tablet_manager.cc:1145] Deleting tablet f63d69d04b494a848653636c9e259105
0110 02:51:32.615584 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.696899 (+ 81315us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.766774 (+ 69875us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.777701 (+ 10927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152127}
I20250110 02:51:32.779361 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f63d69d04b494a848653636c9e259105 (table default.Table_06ee6a222f9f4105a670d8943bf209bf [id=6e40ed00a4164029bcf02876efd12310]) successfully deleted
I20250110 02:51:32.772660 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 235) took 4348 ms. Trace:
I20250110 02:51:32.779713 2357 rpcz_store.cc:276] 0110 02:51:28.423751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.426022 (+ 2271us) service_pool.cc:225] Handling call
0110 02:51:32.772586 (+4346564us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:32.772594 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.772646 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4343905}
I20250110 02:51:32.780031 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 231) took 4356 ms. Trace:
I20250110 02:51:32.780095 2357 rpcz_store.cc:276] 0110 02:51:28.423432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.426473 (+ 3041us) service_pool.cc:225] Handling call
0110 02:51:32.779939 (+4353466us) ts_tablet_manager.cc:1145] Deleting tablet 95b14d7592024a34baf7756d92e546d9
0110 02:51:32.779948 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.780017 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4351172}
I20250110 02:51:32.780447 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 237) took 4297 ms. Trace:
I20250110 02:51:32.780481 3150 tablet.cc:1721] T d637597047ab4738a17eca2f5409459a P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:32.780514 2357 rpcz_store.cc:276] 0110 02:51:28.483207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.483256 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:32.780346 (+4297090us) ts_tablet_manager.cc:1145] Deleting tablet 1d634dd0c2044f91b304fe3737330d88
0110 02:51:32.780359 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.780434 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4296904}
I20250110 02:51:32.780709 2357 tablet_replica.cc:331] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.780872 2357 raft_consensus.cc:2238] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.781013 2357 raft_consensus.cc:2267] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:32.781606 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because the tablet was not found. No further retry: Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:32.781693 2357 ts_tablet_manager.cc:1905] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:32.781837 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 95b14d7592024a34baf7756d92e546d9 because the tablet was not found. No further retry: Not found: Tablet not found: 95b14d7592024a34baf7756d92e546d9
W20250110 02:51:32.782064 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1d634dd0c2044f91b304fe3737330d88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d634dd0c2044f91b304fe3737330d88 already in progress: creating tablet
I20250110 02:51:32.786329 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 313) took 5323 ms. Trace:
I20250110 02:51:32.786700 2224 rpcz_store.cc:276] 0110 02:51:27.462540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.463596 (+ 1056us) service_pool.cc:225] Handling call
0110 02:51:32.786247 (+5322651us) ts_tablet_manager.cc:1145] Deleting tablet cf711522866d46439d7ee69aa63d7f39
0110 02:51:32.786257 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.786313 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5322493}
I20250110 02:51:32.788908 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:32.789381 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:32.789847 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
W20250110 02:51:32.790795 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cf711522866d46439d7ee69aa63d7f39 because the tablet was not found. No further retry: Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39
I20250110 02:51:32.791592 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:32.792498 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:32.792702 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 340) took 5082 ms. Trace:
I20250110 02:51:32.792883 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:32.793000 2224 rpcz_store.cc:276] 0110 02:51:27.710266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.710308 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:32.792626 (+5082318us) ts_tablet_manager.cc:1145] Deleting tablet e744cdc1fcb94d46b48b684763ecf1bf
0110 02:51:32.792634 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.792688 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5082119}
I20250110 02:51:32.793140 1456 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
W20250110 02:51:32.793599 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e744cdc1fcb94d46b48b684763ecf1bf because the tablet was not found. No further retry: Not found: Tablet not found: e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:32.795320 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 341) took 5083 ms. Trace:
I20250110 02:51:32.795487 2224 rpcz_store.cc:276] 0110 02:51:27.711244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.711377 (+ 133us) service_pool.cc:225] Handling call
0110 02:51:32.794536 (+5083159us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:32.794543 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.794593 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5082962}
I20250110 02:51:32.795818 1188 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
W20250110 02:51:32.795997 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because the tablet was not found. No further retry: Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:32.796300 2224 tablet_replica.cc:331] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.796655 2224 raft_consensus.cc:2238] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.797252 1188 tablet_bootstrap.cc:654] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.797657 3152 raft_consensus.cc:491] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.797755 3152 raft_consensus.cc:513] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.797902 2906 raft_consensus.cc:491] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.797979 2906 raft_consensus.cc:513] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.798331 2906 leader_election.cc:290] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.798941 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.798970 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.799132 768 raft_consensus.cc:2463] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:32.799571 994 leader_election.cc:343] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.797761 2224 raft_consensus.cc:2267] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.800686 996 leader_election.cc:304] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.800915 3152 leader_election.cc:290] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.801023 3152 raft_consensus.cc:2798] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.801072 3152 raft_consensus.cc:491] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.801126 3152 raft_consensus.cc:3054] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.802464 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.802659 2224 ts_tablet_manager.cc:1905] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.803474 1184 tablet_bootstrap.cc:492] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.803646 1184 ts_tablet_manager.cc:1397] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
W20250110 02:51:32.802923 994 leader_election.cc:343] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.804001 3152 raft_consensus.cc:513] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.804458 3152 leader_election.cc:290] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.804700 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.803452 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.805100 772 raft_consensus.cc:2463] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
W20250110 02:51:32.805450 994 leader_election.cc:343] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.806756 996 leader_election.cc:304] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.806774 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.807019 772 raft_consensus.cc:3054] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.807046 3152 raft_consensus.cc:2798] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.807102 3152 raft_consensus.cc:491] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.807175 3152 raft_consensus.cc:3054] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.809525 3152 raft_consensus.cc:513] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.810298 3158 tablet.cc:1721] T d637597047ab4738a17eca2f5409459a P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:32.810397 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.810537 768 raft_consensus.cc:3054] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.812532 772 raft_consensus.cc:2463] T 4bd2283e7ed7456dae466a8582587eb7 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.812588 768 raft_consensus.cc:2463] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.813099 996 leader_election.cc:304] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.813316 2906 raft_consensus.cc:2798] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.813421 2906 raft_consensus.cc:695] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.813432 996 leader_election.cc:304] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.813640 2906 consensus_queue.cc:237] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.813748 2738 raft_consensus.cc:2798] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.813872 2738 raft_consensus.cc:695] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.814049 2738 consensus_queue.cc:237] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.815155 1181 tablet_bootstrap.cc:492] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.815275 1181 ts_tablet_manager.cc:1397] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20250110 02:51:32.815841 1181 raft_consensus.cc:357] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.816416 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5ac15f4e774b7e854e4c5b4737b591" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.804432 1184 raft_consensus.cc:357] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.816856 1184 raft_consensus.cc:383] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.816967 1184 raft_consensus.cc:738] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.817278 1184 consensus_queue.cc:260] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.817926 1184 ts_tablet_manager.cc:1428] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250110 02:51:32.818289 1184 tablet_bootstrap.cc:492] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.820080 1184 tablet_bootstrap.cc:654] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:32.816920 994 leader_election.cc:343] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.818660 3152 leader_election.cc:290] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.826229 1188 tablet_bootstrap.cc:492] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.826617 1188 ts_tablet_manager.cc:1397] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20250110 02:51:32.827322 1188 raft_consensus.cc:357] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.823061 1181 raft_consensus.cc:383] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.831264 1181 raft_consensus.cc:738] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.831753 1181 consensus_queue.cc:260] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.833565 1181 ts_tablet_manager.cc:1428] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250110 02:51:32.834230 1181 tablet_bootstrap.cc:492] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.834311 1188 raft_consensus.cc:383] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.834470 1188 raft_consensus.cc:738] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.834739 1188 consensus_queue.cc:260] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.840653 1181 tablet_bootstrap.cc:654] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.843245 3149 tablet.cc:1721] T d637597047ab4738a17eca2f5409459a P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:32.846937 3162 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d8d40556dc348e8aa8255d116251a10)
I20250110 02:51:32.847116 3162 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fca9a92f47145fcaaf4f65038d6f558" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35975 } } peers { permanent_uuid: "8bc934cc708045e9858ee8f8e64660cd" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 44449 } } peers { permanent_uuid: "5d8d40556dc348e8aa8255d116251a10" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 39025 } }
I20250110 02:51:32.847827 3162 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8fca9a92f47145fcaaf4f65038d6f558 (127.0.104.62:35975), 5d8d40556dc348e8aa8255d116251a10 (127.0.104.60:39025)
I20250110 02:51:32.850744 1478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 864 } } ignore_live_leader: false dest_uuid: "5d8d40556dc348e8aa8255d116251a10" is_pre_election: true
I20250110 02:51:32.850776 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bc934cc708045e9858ee8f8e64660cd" candidate_term: 5 candidate_status { last_received { term: 4 index: 864 } } ignore_live_leader: false dest_uuid: "8fca9a92f47145fcaaf4f65038d6f558" is_pre_election: true
I20250110 02:51:32.852006 1188 ts_tablet_manager.cc:1428] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20250110 02:51:32.858276 2908 raft_consensus.cc:491] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.858424 2908 raft_consensus.cc:513] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.859061 2908 leader_election.cc:290] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.860668 2310 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8bc934cc708045e9858ee8f8e64660cd; no voters: 5d8d40556dc348e8aa8255d116251a10, 8fca9a92f47145fcaaf4f65038d6f558
I20250110 02:51:32.861303 3162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 02:51:32.861788 1188 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.866061 914 raft_consensus.cc:3054] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.866223 911 raft_consensus.cc:3054] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.866760 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.867486 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718d3add9d0a4e4b8633f2c977e316e6" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.867908 853 leader_election.cc:304] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.870831 1188 tablet_bootstrap.cc:654] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.872934 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:32.873385 2906 raft_consensus.cc:491] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.873519 2906 raft_consensus.cc:513] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.874011 2906 leader_election.cc:290] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.874444 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.874660 772 raft_consensus.cc:2463] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.874958 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.875468 996 leader_election.cc:304] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:32.875691 994 leader_election.cc:343] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.875820 2906 raft_consensus.cc:2798] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.875885 2906 raft_consensus.cc:491] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.875950 2906 raft_consensus.cc:3054] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.878226 2906 raft_consensus.cc:513] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.878906 2906 leader_election.cc:290] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.879380 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.879587 768 raft_consensus.cc:3054] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.875447 1448 catalog_manager.cc:5526] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98336b684b41407da8a1949e30cd82e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "98336b684b41407da8a1949e30cd82e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:32.880434 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:32.881237 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:32.882122 768 raft_consensus.cc:2463] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.882283 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:32.882783 996 leader_election.cc:304] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.882997 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:32.883337 2738 raft_consensus.cc:2798] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.883445 2738 raft_consensus.cc:695] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.883296 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:32.883661 2738 consensus_queue.cc:237] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.884028 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
W20250110 02:51:32.884258 994 leader_election.cc:343] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.884843 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d637597047ab4738a17eca2f5409459a (table default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534]). Expected version 1 got 0
I20250110 02:51:32.888114 1188 tablet_bootstrap.cc:492] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.889225 1188 ts_tablet_manager.cc:1397] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250110 02:51:32.889901 1188 raft_consensus.cc:357] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.890559 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:32.890616 3152 raft_consensus.cc:491] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.890723 3152 raft_consensus.cc:513] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.891105 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:32.891557 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:32.892007 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:32.892051 3152 leader_election.cc:290] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.892069 1188 raft_consensus.cc:383] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.892123 1188 raft_consensus.cc:738] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.892686 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:32.892665 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.893005 772 raft_consensus.cc:2463] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:32.892498 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:32.893513 996 leader_election.cc:304] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:32.893786 1188 consensus_queue.cc:260] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:32.893985 994 leader_election.cc:343] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.894289 3152 raft_consensus.cc:2798] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:32.894356 3152 raft_consensus.cc:491] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:32.894439 3152 raft_consensus.cc:3054] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.894429 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet bfd120d8edf14151b7ebed3b74e2b399 (table default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701]). Expected version 1 got 0
I20250110 02:51:32.894906 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_614d56e4a7544c27aa6f21776b2138f3 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 3 got 0
I20250110 02:51:32.895727 1188 ts_tablet_manager.cc:1428] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 02:51:32.895939 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:32.896658 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 9e28d8769f364e10aad54b60fb221f35 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:32.896826 3152 raft_consensus.cc:513] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.897166 1188 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.897445 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:32.898022 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:32.898228 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:32.898490 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:32.898590 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:32.899080 994 leader_election.cc:343] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.899729 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:32.899894 772 raft_consensus.cc:3054] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:32.904755 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:32.905525 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4f1d7940a1574af1804d5f5139ca7494 (table default.Table_b4223b31a5974abb9725bceb8026a8ae [id=4e2fb95ceb8c43fc920b1cd9e73557df]). Expected version 1 got 0
I20250110 02:51:32.902029 772 raft_consensus.cc:2463] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:32.907711 996 leader_election.cc:304] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:32.908100 2289 raft_consensus.cc:2798] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:32.908221 2289 raft_consensus.cc:695] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:32.908404 2289 consensus_queue.cc:237] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.909080 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:32.909690 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:32.911265 2224 ts_tablet_manager.cc:1918] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:32.915751 2691 raft_consensus.cc:491] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.915863 2691 raft_consensus.cc:513] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.916370 2691 leader_election.cc:290] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.917027 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:32.920892 1188 tablet_bootstrap.cc:654] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.923702 2199 raft_consensus.cc:491] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.923810 2199 raft_consensus.cc:513] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.924345 2199 leader_election.cc:290] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:32.925201 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9497c76dcffa45268f705ca6423db41c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:32.925663 709 leader_election.cc:343] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.916694 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ffe7bf29a4cbcb4faef6f3b2931ec" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.919111 1181 tablet_bootstrap.cc:492] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.927011 1181 ts_tablet_manager.cc:1397] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.000s
I20250110 02:51:32.927635 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:32.927671 1181 raft_consensus.cc:357] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.927922 1181 raft_consensus.cc:383] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.928014 1181 raft_consensus.cc:738] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.928231 1181 consensus_queue.cc:260] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.928369 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:32.928793 1181 ts_tablet_manager.cc:1428] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:32.929052 1181 tablet_bootstrap.cc:492] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:32.929558 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:32.929809 853 leader_election.cc:304] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.930272 2691 raft_consensus.cc:2743] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.924703 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9497c76dcffa45268f705ca6423db41c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:32.924022 3152 leader_election.cc:290] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:32.930735 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
W20250110 02:51:32.930859 711 leader_election.cc:343] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:32.930984 711 leader_election.cc:304] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:32.931197 2199 raft_consensus.cc:2743] T 9497c76dcffa45268f705ca6423db41c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.933511 1188 tablet_bootstrap.cc:492] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.935026 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:32.935317 1188 ts_tablet_manager.cc:1397] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20250110 02:51:32.935612 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:32.936350 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:32.940702 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 3aee71c51a96482fbb527576867bbaa9 (table default.Table_a99fe640359046aa8ff288db7a687676 [id=c9a367ed70c742a18d7dc76e72b37300]). Expected version 1 got 0
I20250110 02:51:32.942777 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:32.943277 1448 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:32.943943 1456 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_7cd682c4f72745b5aa221478659d3350 [id=28ef0a56c1054854a22896fb038a2aac] alter complete (version 1)
I20250110 02:51:32.946818 2224 log.cc:1198] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:32.947207 2224 ts_tablet_manager.cc:1939] T 80186dd7d1a240068584b64ce9e9b03f P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:32.947265 1444 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_5f85d34e73f44a2a9eec8879fda3d338 [id=7e19567637ca466c8df85f6e0aaee0bc] alter complete (version 1)
I20250110 02:51:32.947881 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 342) took 5236 ms. Trace:
I20250110 02:51:32.947960 2224 rpcz_store.cc:276] 0110 02:51:27.711422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.711460 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:32.796262 (+5084802us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:32.796270 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.908137 (+111867us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.911197 (+ 3060us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.947831 (+ 36634us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5084607}
I20250110 02:51:32.948287 2224 tablet_replica.cc:331] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:32.948387 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 80186dd7d1a240068584b64ce9e9b03f (table default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]) successfully deleted
I20250110 02:51:32.948474 2224 raft_consensus.cc:2238] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.948648 2224 raft_consensus.cc:2267] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.950899 1443 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 169) took 3906 ms. Trace:
I20250110 02:51:32.951040 1443 rpcz_store.cc:276] 0110 02:51:29.044188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.396127 (+2351939us) service_pool.cc:225] Handling call
0110 02:51:31.396550 (+ 423us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:31.396748 (+ 198us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:31.396936 (+ 188us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:31.397016 (+ 80us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:31.402761 (+ 5745us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.402765 (+ 4us) write_op.cc:454] Released schema lock
0110 02:51:31.402816 (+ 51us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:32.950820 (+1548004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.397600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.397692 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.397698 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:31.397699 (+ 1us) tablet.cc:659] Decoding operations
0110 02:51:31.397796 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.397808 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.397814 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:31.397870 (+ 56us) tablet.cc:698] Row locks acquired
0110 02:51:31.397873 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:31.397914 (+ 41us) write_op.cc:270] Start()
0110 02:51:31.397940 (+ 26us) write_op.cc:276] Timestamp: P: 1736477491397854 usec, L: 0
0110 02:51:31.397944 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.398078 (+ 134us) log.cc:844] Serialized 405 byte log entry
0110 02:51:31.400886 (+ 2808us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.402274 (+ 1388us) write_op.cc:301] APPLY: starting
0110 02:51:31.402286 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.402313 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.402333 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.402366 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.402414 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:31.402420 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:31.402474 (+ 54us) log.cc:844] Serialized 23 byte log entry
0110 02:51:31.402518 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.402530 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:31.402535 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1345,"apply.run_cpu_time_us":306,"apply.run_wall_time_us":305,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":581,"raft.queue_time_us":54,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":304,"replication_time_us":2919}]]}
I20250110 02:51:32.953814 1457 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_6aaaf7ba453a4e74a3a55367388c0584 [id=ee9fe75dbd374159bb7a40bdc3576d06] alter complete (version 3)
I20250110 02:51:32.936399 1188 raft_consensus.cc:357] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.954557 1188 raft_consensus.cc:383] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.954607 1188 raft_consensus.cc:738] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.955072 2357 ts_tablet_manager.cc:1918] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:32.955154 2357 log.cc:1198] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/2f0f754966c04074a568c05063591990
I20250110 02:51:32.955382 2357 ts_tablet_manager.cc:1939] T 2f0f754966c04074a568c05063591990 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:32.956043 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 238) took 4463 ms. Trace:
I20250110 02:51:32.958678 1188 consensus_queue.cc:260] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:32.959141 2908 raft_consensus.cc:2743] T 718d3add9d0a4e4b8633f2c977e316e6 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.959236 1188 ts_tablet_manager.cc:1428] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.023s user 0.000s sys 0.000s
I20250110 02:51:32.959499 1188 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.960851 1188 tablet_bootstrap.cc:654] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.963538 1181 tablet_bootstrap.cc:654] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.965723 1184 tablet_bootstrap.cc:492] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:32.965822 1184 ts_tablet_manager.cc:1397] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.007s
I20250110 02:51:32.966363 1184 raft_consensus.cc:357] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.966536 1184 raft_consensus.cc:383] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.966575 1184 raft_consensus.cc:738] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.966753 1184 consensus_queue.cc:260] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.956110 2357 rpcz_store.cc:276] 0110 02:51:28.492414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.492461 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:32.780676 (+4288215us) ts_tablet_manager.cc:1145] Deleting tablet 2f0f754966c04074a568c05063591990
0110 02:51:32.780683 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.784822 (+ 4139us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.954977 (+170155us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:32.956024 (+ 1047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4288061}
I20250110 02:51:32.971871 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 239) took 4387 ms. Trace:
I20250110 02:51:32.971949 2357 rpcz_store.cc:276] 0110 02:51:28.584653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.584718 (+ 65us) service_pool.cc:225] Handling call
0110 02:51:32.971762 (+4387044us) ts_tablet_manager.cc:1145] Deleting tablet 1d064a7e8ea34f18b42407903daf7197
0110 02:51:32.971771 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.971857 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4354328}
I20250110 02:51:32.972909 2357 tablet_replica.cc:331] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:32.973081 2357 raft_consensus.cc:2238] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:32.973243 2357 raft_consensus.cc:2267] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:32.974105 1184 ts_tablet_manager.cc:1428] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250110 02:51:32.974344 1184 tablet_bootstrap.cc:492] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:32.976310 1188 tablet_bootstrap.cc:492] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:32.981114 1188 ts_tablet_manager.cc:1397] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20250110 02:51:32.982367 1188 raft_consensus.cc:357] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.983819 1188 raft_consensus.cc:383] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.983985 1188 raft_consensus.cc:738] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.984985 2357 ts_tablet_manager.cc:1905] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:32.985206 2908 raft_consensus.cc:491] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:32.985850 2908 raft_consensus.cc:513] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.986385 2908 leader_election.cc:290] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:32.987322 853 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
W20250110 02:51:32.987625 853 leader_election.cc:343] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:32.987704 853 leader_election.cc:304] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:32.988008 2908 raft_consensus.cc:2743] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:32.988395 1188 consensus_queue.cc:260] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.990065 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 2f0f754966c04074a568c05063591990 (table default.Table_424f77049b1645eb850dfb8e41db29bb [id=eb7db3bbedcc4f5287548c49a5f5bde9]) successfully deleted
W20250110 02:51:32.990238 994 consensus_peers.cc:487] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:32.990348 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1d064a7e8ea34f18b42407903daf7197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d064a7e8ea34f18b42407903daf7197 already in progress: creating tablet
W20250110 02:51:32.991319 853 consensus_peers.cc:487] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:32.989409 1188 ts_tablet_manager.cc:1428] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250110 02:51:32.992218 1188 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:32.993894 1188 tablet_bootstrap.cc:654] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:32.994534 1444 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_796db82db8ec4b7ebb6d3d3ab676114f [id=f84d6095952546dd94aedb2f33aa490e] alter complete (version 1)
I20250110 02:51:32.997998 1181 tablet_bootstrap.cc:492] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:32.998170 1181 ts_tablet_manager.cc:1397] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.001s
I20250110 02:51:32.999219 1181 raft_consensus.cc:357] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:32.999423 1181 raft_consensus.cc:383] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:32.999444 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:32.999495 1181 raft_consensus.cc:738] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:32.999732 1181 consensus_queue.cc:260] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.003432 2357 ts_tablet_manager.cc:1918] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:33.003515 2357 log.cc:1198] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f0d4d6e02b1d4334a9993d320217f1f8
I20250110 02:51:33.003814 2357 ts_tablet_manager.cc:1939] T f0d4d6e02b1d4334a9993d320217f1f8 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.005249 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.005458 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.005627 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.005918 1181 ts_tablet_manager.cc:1428] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:33.006186 1181 tablet_bootstrap.cc:492] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.008242 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e4af23720048fe98b3231ba402d45c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.008788 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.008836 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet b382d634db9642df8e00c5ccaa5787a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.009054 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.009238 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.009253 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d970e2fe72b4727968ca605b3e772cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.010008 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb4e06e40c43438882e1eb622d788e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.010244 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9289de0630804f9aa60fdaf5f3f15f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.015698 1181 tablet_bootstrap.cc:654] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.016218 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.000671 2352 ts_tablet_manager.cc:1918] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.016474 2352 log.cc:1198] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:33.016772 2352 ts_tablet_manager.cc:1939] T 04e05a7554a14805bdcbd1824a5bd9d5 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.017383 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.019115 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 265) took 6676 ms. Trace:
I20250110 02:51:33.019299 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.020210 2352 rpcz_store.cc:276] 0110 02:51:26.342231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.342272 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:32.702503 (+6360231us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:32.702513 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.993494 (+290981us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.000589 (+ 7095us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.019097 (+ 18508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6360073}
I20250110 02:51:33.021804 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 266) took 6679 ms. Trace:
I20250110 02:51:33.022404 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 04e05a7554a14805bdcbd1824a5bd9d5 (table default.Table_3c02731f843a4cc3b13b54dee5b9e7f7 [id=47fcabfd021b4a86a641bccce0d6f9a5]) successfully deleted
I20250110 02:51:33.022408 2352 rpcz_store.cc:276] 0110 02:51:26.342571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.342604 (+ 33us) service_pool.cc:225] Handling call
0110 02:51:33.021716 (+6679112us) ts_tablet_manager.cc:1145] Deleting tablet da8f247c862e4cb68452e230857ee6b0
0110 02:51:33.021723 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.021789 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6678991}
W20250110 02:51:33.022791 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet da8f247c862e4cb68452e230857ee6b0 because the tablet was not found. No further retry: Not found: Tablet not found: da8f247c862e4cb68452e230857ee6b0
I20250110 02:51:33.023028 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 240) took 4433 ms. Trace:
I20250110 02:51:33.023049 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.023120 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 267) took 6680 ms. Trace:
I20250110 02:51:33.023195 2352 rpcz_store.cc:276] 0110 02:51:26.342887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.344198 (+ 1311us) service_pool.cc:225] Handling call
0110 02:51:33.023043 (+6678845us) ts_tablet_manager.cc:1145] Deleting tablet 7d4422eca5604d149e73c2a669d07af0
0110 02:51:33.023054 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.023105 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6678715}
I20250110 02:51:33.023181 2357 rpcz_store.cc:276] 0110 02:51:28.589595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.589644 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:32.972861 (+4383217us) ts_tablet_manager.cc:1145] Deleting tablet f0d4d6e02b1d4334a9993d320217f1f8
0110 02:51:32.972870 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:32.988149 (+ 15279us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.003354 (+ 15205us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.023007 (+ 19653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355386}
W20250110 02:51:33.023864 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7d4422eca5604d149e73c2a669d07af0 because the tablet was not found. No further retry: Not found: Tablet not found: 7d4422eca5604d149e73c2a669d07af0
I20250110 02:51:33.024259 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09]) successfully deleted
I20250110 02:51:33.024382 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 65285bce60fd4015b085deb7442eea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.024722 2357 tablet_replica.cc:331] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.025806 2357 raft_consensus.cc:2238] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.025164 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 269) took 6680 ms. Trace:
I20250110 02:51:33.027940 2352 rpcz_store.cc:276] 0110 02:51:26.344826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:26.344868 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:33.025083 (+6680215us) ts_tablet_manager.cc:1145] Deleting tablet 5f229f546a8d4a09b93f8b46c893d14a
0110 02:51:33.025090 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.025138 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6680083}
I20250110 02:51:33.028124 2357 raft_consensus.cc:2267] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.028895 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 285) took 5353 ms. Trace:
I20250110 02:51:33.028971 2352 rpcz_store.cc:276] 0110 02:51:27.675179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.675239 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:33.028824 (+5353585us) ts_tablet_manager.cc:1145] Deleting tablet 8c3ce95413df40369ee599176f91b892
0110 02:51:33.028832 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.028879 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5353352}
I20250110 02:51:33.030153 2357 ts_tablet_manager.cc:1905] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.030334 1188 tablet_bootstrap.cc:492] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
W20250110 02:51:33.030398 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8c3ce95413df40369ee599176f91b892 because the tablet was not found. No further retry: Not found: Tablet not found: 8c3ce95413df40369ee599176f91b892
I20250110 02:51:33.030973 1188 ts_tablet_manager.cc:1397] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20250110 02:51:33.031757 1188 raft_consensus.cc:357] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.032178 1188 raft_consensus.cc:383] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.032284 1188 raft_consensus.cc:738] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.032095 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 286) took 5334 ms. Trace:
I20250110 02:51:33.031136 2199 raft_consensus.cc:491] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.032482 2352 rpcz_store.cc:276] 0110 02:51:27.697839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.698079 (+ 240us) service_pool.cc:225] Handling call
0110 02:51:33.032010 (+5333931us) ts_tablet_manager.cc:1145] Deleting tablet f63d69d04b494a848653636c9e259105
0110 02:51:33.032017 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.032081 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5333683}
I20250110 02:51:33.032625 2199 raft_consensus.cc:513] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.032809 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f63d69d04b494a848653636c9e259105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f63d69d04b494a848653636c9e259105 already in progress: creating tablet
I20250110 02:51:33.032840 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 287) took 5330 ms. Trace:
I20250110 02:51:33.032896 2352 rpcz_store.cc:276] 0110 02:51:27.702755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.702813 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:33.032776 (+5329963us) ts_tablet_manager.cc:1145] Deleting tablet 971f97a3be9b49958b1218a275e126d4
0110 02:51:33.032783 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.032827 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5329795}
I20250110 02:51:33.033226 2199 leader_election.cc:290] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.033638 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975136880034808b58276e06fbd94f9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:33.033995 711 leader_election.cc:343] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.033205 1188 consensus_queue.cc:260] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.034529 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c975136880034808b58276e06fbd94f9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.034881 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 288) took 5332 ms. Trace:
W20250110 02:51:33.035070 709 leader_election.cc:343] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.035156 709 leader_election.cc:304] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.035421 2352 rpcz_store.cc:276] 0110 02:51:27.702842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.702881 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:33.034817 (+5331936us) ts_tablet_manager.cc:1145] Deleting tablet 5e183122d13a4c38988bf2fe6f7a3694
0110 02:51:33.034825 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.034867 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5331770}
I20250110 02:51:33.035513 2199 raft_consensus.cc:2743] T c975136880034808b58276e06fbd94f9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.035705 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 290) took 5332 ms. Trace:
I20250110 02:51:33.035759 2352 rpcz_store.cc:276] 0110 02:51:27.703474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.703506 (+ 32us) service_pool.cc:225] Handling call
0110 02:51:33.035613 (+5332107us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:33.035627 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.035692 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5331986}
I20250110 02:51:33.036013 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 291) took 5332 ms. Trace:
I20250110 02:51:33.036063 2352 rpcz_store.cc:276] 0110 02:51:27.703775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.703809 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:33.035930 (+5332121us) ts_tablet_manager.cc:1145] Deleting tablet c28f191afa2a43608bbd74322bec97a6
0110 02:51:33.035936 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.036002 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5332001}
I20250110 02:51:33.035362 1188 ts_tablet_manager.cc:1428] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20250110 02:51:33.037755 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 971f97a3be9b49958b1218a275e126d4 because the tablet was not found. No further retry: Not found: Tablet not found: 971f97a3be9b49958b1218a275e126d4
I20250110 02:51:33.038458 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 292) took 5334 ms. Trace:
I20250110 02:51:33.038523 2352 rpcz_store.cc:276] 0110 02:51:27.704067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.704099 (+ 32us) service_pool.cc:225] Handling call
0110 02:51:33.036221 (+5332122us) ts_tablet_manager.cc:1145] Deleting tablet 2e25f9643ac0476a90c93fb9aeb921a3
0110 02:51:33.038345 (+ 2124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.038441 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5332011}
W20250110 02:51:33.038775 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5e183122d13a4c38988bf2fe6f7a3694 because the tablet was not found. No further retry: Not found: Tablet not found: 5e183122d13a4c38988bf2fe6f7a3694
I20250110 02:51:33.038828 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 293) took 5334 ms. Trace:
I20250110 02:51:33.038893 2352 rpcz_store.cc:276] 0110 02:51:27.704361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.704394 (+ 33us) service_pool.cc:225] Handling call
0110 02:51:33.038731 (+5334337us) ts_tablet_manager.cc:1145] Deleting tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8
0110 02:51:33.038740 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.038815 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5334228}
W20250110 02:51:33.038980 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5ae4877bba834dbb96ec9675e2aaedd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae4877bba834dbb96ec9675e2aaedd0 already in progress: creating tablet
W20250110 02:51:33.039160 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c28f191afa2a43608bbd74322bec97a6 already in progress: creating tablet
I20250110 02:51:33.039173 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 294) took 5334 ms. Trace:
I20250110 02:51:33.039233 2352 rpcz_store.cc:276] 0110 02:51:27.704636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.704667 (+ 31us) service_pool.cc:225] Handling call
0110 02:51:33.039090 (+5334423us) ts_tablet_manager.cc:1145] Deleting tablet a7397e41f7374e9fb1621dbcf30af703
0110 02:51:33.039099 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.039159 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5334315}
W20250110 02:51:33.039343 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because the tablet was not found. No further retry: Not found: Tablet not found: 2e25f9643ac0476a90c93fb9aeb921a3
I20250110 02:51:33.039494 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 295) took 5334 ms. Trace:
I20250110 02:51:33.039553 2352 rpcz_store.cc:276] 0110 02:51:27.704915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.704949 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:33.039405 (+5334456us) ts_tablet_manager.cc:1145] Deleting tablet 82bd4307e5fe4f61b9308005cf3446e1
0110 02:51:33.039413 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.039477 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5334344}
I20250110 02:51:33.039844 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 296) took 5334 ms. Trace:
W20250110 02:51:33.040112 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a8cbed1ba6f04fd9a43a8cd6b7adc7b8 because the tablet was not found. No further retry: Not found: Tablet not found: a8cbed1ba6f04fd9a43a8cd6b7adc7b8
W20250110 02:51:33.040282 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a7397e41f7374e9fb1621dbcf30af703 because the tablet was not found. No further retry: Not found: Tablet not found: a7397e41f7374e9fb1621dbcf30af703
W20250110 02:51:33.040452 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82bd4307e5fe4f61b9308005cf3446e1 already in progress: creating tablet
I20250110 02:51:33.040715 2352 rpcz_store.cc:276] 0110 02:51:27.705205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.705237 (+ 32us) service_pool.cc:225] Handling call
0110 02:51:33.039755 (+5334518us) ts_tablet_manager.cc:1145] Deleting tablet fc1324eeac9446279304f442e2fad488
0110 02:51:33.039762 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.039833 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5334411}
I20250110 02:51:33.041227 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 297) took 5335 ms. Trace:
W20250110 02:51:33.041378 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
I20250110 02:51:33.043339 2352 rpcz_store.cc:276] 0110 02:51:27.705469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.705499 (+ 30us) service_pool.cc:225] Handling call
0110 02:51:33.041138 (+5335639us) ts_tablet_manager.cc:1145] Deleting tablet bc8444c1381a45d6b2fee30da971bd47
0110 02:51:33.041153 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.041214 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5335517}
I20250110 02:51:33.040110 1188 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.044919 1188 tablet_bootstrap.cc:654] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.045635 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 298) took 5339 ms. Trace:
I20250110 02:51:33.045746 2352 rpcz_store.cc:276] 0110 02:51:27.705772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.705806 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:33.045568 (+5339762us) ts_tablet_manager.cc:1145] Deleting tablet 60e534bf66f044b88d8cda0f3a1228f4
0110 02:51:33.045575 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.045620 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5339647}
I20250110 02:51:33.046169 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 299) took 5327 ms. Trace:
I20250110 02:51:33.046277 2352 rpcz_store.cc:276] 0110 02:51:27.718764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.718827 (+ 63us) service_pool.cc:225] Handling call
0110 02:51:33.046069 (+5327242us) ts_tablet_manager.cc:1145] Deleting tablet a7d4f03a8c66446ea7c6d178cc7ef80a
0110 02:51:33.046077 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.046139 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327069}
I20250110 02:51:33.046734 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 300) took 5327 ms. Trace:
I20250110 02:51:33.046839 2352 rpcz_store.cc:276] 0110 02:51:27.719132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.719172 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:33.046648 (+5327476us) ts_tablet_manager.cc:1145] Deleting tablet 213939cf13354ce380fb7cda5b38676f
0110 02:51:33.046657 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.046720 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327343}
I20250110 02:51:33.047101 1181 tablet_bootstrap.cc:492] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.047243 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 301) took 5327 ms. Trace:
I20250110 02:51:33.047273 1181 ts_tablet_manager.cc:1397] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.004s
I20250110 02:51:33.047355 2352 rpcz_store.cc:276] 0110 02:51:27.719456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.719495 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:33.047179 (+5327684us) ts_tablet_manager.cc:1145] Deleting tablet 2ef72e05f8ed4d46b759c7a04497653b
0110 02:51:33.047188 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.047231 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327556}
I20250110 02:51:33.047744 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 302) took 5327 ms. Trace:
I20250110 02:51:33.047890 1181 raft_consensus.cc:357] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.048063 1181 raft_consensus.cc:383] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.048427 1181 raft_consensus.cc:738] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.048671 1181 consensus_queue.cc:260] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.048826 2352 rpcz_store.cc:276] 0110 02:51:27.719808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.719861 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:33.047680 (+5327819us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:33.047687 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.047730 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327684}
I20250110 02:51:33.048972 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049463 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edb4e06e40c43438882e1eb622d788e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049540 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049646 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049818 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049966 1181 ts_tablet_manager.cc:1428] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250110 02:51:33.049962 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050094 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050168 1181 tablet_bootstrap.cc:492] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.050230 748 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6b2654f86e4686b8a3034de02d11bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050357 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050503 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e848586bec574a529caaf5e498db79cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050520 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 303) took 5330 ms. Trace:
I20250110 02:51:33.050583 2352 rpcz_store.cc:276] 0110 02:51:27.720467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.720507 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:33.050459 (+5329952us) ts_tablet_manager.cc:1145] Deleting tablet 344184e1c98d422095cb254f5b2bc792
0110 02:51:33.050467 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.050508 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5329825}
I20250110 02:51:33.050623 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d6acae633eb14e5db87b1e36a069422c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050822 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d064a7e8ea34f18b42407903daf7197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.050860 2352 tablet_replica.cc:331] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.050953 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0f754966c04074a568c05063591990 with delete_type TABLET_DATA_DELETED (replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051013 2352 raft_consensus.cc:2238] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.051080 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d634dd0c2044f91b304fe3737330d88 with delete_type TABLET_DATA_DELETED (replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051121 2352 raft_consensus.cc:2267] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.051219 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051339 748 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051448 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.049818 739 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051586 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 6555b0a4491046bf8f5cd4ee406c3955 with delete_type TABLET_DATA_DELETED (replaced by tablet b5f162de87044005bdafe93292ff71b8) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051709 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.051873 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b937188bf2949af8f54bade4e0ad984 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e53b838be846c7b3ac44dc75aa3730) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052026 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052094 2352 ts_tablet_manager.cc:1905] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.052173 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052340 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e25f9643ac0476a90c93fb9aeb921a3 with delete_type TABLET_DATA_DELETED (replaced by tablet d637597047ab4738a17eca2f5409459a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052487 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052641 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052771 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.052911 746 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.053857 756 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:33.054575 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
W20250110 02:51:33.054826 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 60e534bf66f044b88d8cda0f3a1228f4 because the tablet was not found. No further retry: Not found: Tablet not found: 60e534bf66f044b88d8cda0f3a1228f4
W20250110 02:51:33.055017 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7d4f03a8c66446ea7c6d178cc7ef80a already in progress: creating tablet
W20250110 02:51:33.055184 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 213939cf13354ce380fb7cda5b38676f already in progress: creating tablet
W20250110 02:51:33.055357 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2ef72e05f8ed4d46b759c7a04497653b because the tablet was not found. No further retry: Not found: Tablet not found: 2ef72e05f8ed4d46b759c7a04497653b
W20250110 02:51:33.055507 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because the tablet was not found. No further retry: Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
W20250110 02:51:33.055696 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 344184e1c98d422095cb254f5b2bc792 because the tablet was not found. No further retry: Not found: Tablet not found: 344184e1c98d422095cb254f5b2bc792
I20250110 02:51:33.057205 2357 ts_tablet_manager.cc:1918] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.057298 2357 log.cc:1198] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/071bbce1706c43b6bb690c7de1ead3fe
I20250110 02:51:33.057600 2357 ts_tablet_manager.cc:1939] T 071bbce1706c43b6bb690c7de1ead3fe P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.058292 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 241) took 4462 ms. Trace:
I20250110 02:51:33.058383 2357 rpcz_store.cc:276] 0110 02:51:28.595910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.595966 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:33.024663 (+4428697us) ts_tablet_manager.cc:1145] Deleting tablet 071bbce1706c43b6bb690c7de1ead3fe
0110 02:51:33.024675 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.044493 (+ 19818us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.057114 (+ 12621us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.058276 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4407140}
I20250110 02:51:33.058920 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 071bbce1706c43b6bb690c7de1ead3fe (table default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]) successfully deleted
I20250110 02:51:33.064225 2357 tablet_replica.cc:331] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.064460 2357 raft_consensus.cc:2238] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.064648 2357 raft_consensus.cc:2267] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.065147 2357 ts_tablet_manager.cc:1905] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.066277 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.069370 2199 raft_consensus.cc:491] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.069494 2199 raft_consensus.cc:513] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.070087 2199 leader_election.cc:290] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.070225 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.070461 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.070528 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8f8d0ac49a428fb7cbabb1e27bf34e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:33.071027 711 leader_election.cc:343] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.071137 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8f8d0ac49a428fb7cbabb1e27bf34e" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.071211 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:33.071709 709 leader_election.cc:343] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.071815 709 leader_election.cc:304] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.072202 2199 raft_consensus.cc:2743] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.072369 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.072930 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.073808 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9ead3175140028e584e1ccb4874de with delete_type TABLET_DATA_DELETED (replaced by tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.074771 1188 tablet_bootstrap.cc:492] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.074898 1188 ts_tablet_manager.cc:1397] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20250110 02:51:33.075454 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:11 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.075666 1188 raft_consensus.cc:357] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.075826 1188 raft_consensus.cc:383] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.075874 1188 raft_consensus.cc:738] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.076086 1188 consensus_queue.cc:260] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.076604 1188 ts_tablet_manager.cc:1428] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:33.078066 1188 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.079126 2357 ts_tablet_manager.cc:1918] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.079280 2357 log.cc:1198] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/bb595013e809480a814cb9c7aed856e6
I20250110 02:51:33.079684 1188 tablet_bootstrap.cc:654] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.080125 2357 ts_tablet_manager.cc:1939] T bb595013e809480a814cb9c7aed856e6 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.082147 2352 ts_tablet_manager.cc:1918] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.082280 2352 log.cc:1198] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:33.082729 2352 ts_tablet_manager.cc:1939] T d51adff7f50c4aec94e1892a50ec36c4 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.083599 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 304) took 5362 ms. Trace:
I20250110 02:51:33.084124 2352 rpcz_store.cc:276] 0110 02:51:27.720792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.720842 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:33.050823 (+5329981us) ts_tablet_manager.cc:1145] Deleting tablet d51adff7f50c4aec94e1892a50ec36c4
0110 02:51:33.050831 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.060368 (+ 9537us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.082040 (+ 21672us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.083583 (+ 1543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5329854}
I20250110 02:51:33.077144 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.076593 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.076071 2224 ts_tablet_manager.cc:1905] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.084861 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 305) took 5363 ms. Trace:
I20250110 02:51:33.084935 2352 rpcz_store.cc:276] 0110 02:51:27.721075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.721120 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:33.084754 (+5363634us) ts_tablet_manager.cc:1145] Deleting tablet 4aa78a9ef5eb443da0c8983a7190440b
0110 02:51:33.084763 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.084847 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5363509}
I20250110 02:51:33.083973 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.085139 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet d51adff7f50c4aec94e1892a50ec36c4 (table default.Table_1fd67f09eeae41fdb559eabdb1d2fc82 [id=a52a8c6bfcc5465bae7f5a2c304a066b]) successfully deleted
I20250110 02:51:33.077656 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.084863 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 940445840db3441598eba57d3ccb8c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.078044 1181 tablet_bootstrap.cc:654] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.085889 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b2163c343a694f5ca04ac9f7610d25e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1df15c10ef2d4bf99af06a860c6f250f) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:33.090830 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:33.090950 3074 catalog_manager.cc:1260] Loaded cluster ID: 2661edda8b194c61b2cd1e6cf1a37395
I20250110 02:51:33.091461 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 306) took 5370 ms. Trace:
I20250110 02:51:33.091791 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.091784 2352 rpcz_store.cc:276] 0110 02:51:27.721377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.721415 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:33.091365 (+5369950us) ts_tablet_manager.cc:1145] Deleting tablet 3fa76357abae4a95b8fb3af47288e43a
0110 02:51:33.091377 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.091444 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5369116}
I20250110 02:51:33.092411 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet c28f191afa2a43608bbd74322bec97a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.092619 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet c5601ca24b8f428e8a51cbeec3664901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.092883 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 55edfa05075b4edcb70ef29e5ee413c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.093015 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.093353 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bd4307e5fe4f61b9308005cf3446e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f015e99f53f48599ac2d7a29220bc39) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.093429 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.093533 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.094027 3074 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: loading cluster ID for follower catalog manager: success
I20250110 02:51:33.095180 2908 raft_consensus.cc:491] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.095396 2908 raft_consensus.cc:513] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.095988 2908 leader_election.cc:290] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:33.096820 853 leader_election.cc:343] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 7e554d00d05b4fa7a14c9b6159f9d726
W20250110 02:51:33.097479 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3fa76357abae4a95b8fb3af47288e43a because the tablet was not found. No further retry: Not found: Tablet not found: 3fa76357abae4a95b8fb3af47288e43a
I20250110 02:51:33.097681 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:33.097434 853 leader_election.cc:343] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 7e554d00d05b4fa7a14c9b6159f9d726
I20250110 02:51:33.097904 853 leader_election.cc:304] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.098253 2908 raft_consensus.cc:2743] T 7e554d00d05b4fa7a14c9b6159f9d726 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.097287 2352 tablet_replica.cc:331] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.098834 2352 raft_consensus.cc:2238] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.099156 2352 raft_consensus.cc:2267] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.099252 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e88d1bbc304b4ebbc849a7d783052e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.101173 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.101281 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.101424 2352 ts_tablet_manager.cc:1905] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.104230 3152 raft_consensus.cc:491] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.104354 3152 raft_consensus.cc:513] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.104920 3152 leader_election.cc:290] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.105502 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.105998 994 leader_election.cc:343] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.110833 853 consensus_peers.cc:487] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d -> Peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Couldn't send request to peer 8899dd1aeb384e4ea4523f5fd225c888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.118934 914 raft_consensus.cc:3054] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.122731 2289 raft_consensus.cc:491] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.122839 2289 raft_consensus.cc:513] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.123342 2289 leader_election.cc:290] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
W20250110 02:51:33.123592 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 130) took 4459 ms (client timeout 1000 ms). Trace:
W20250110 02:51:33.123700 1439 rpcz_store.cc:269] 0110 02:51:28.663954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.718479 (+ 54525us) service_pool.cc:225] Handling call
0110 02:51:28.879053 (+160574us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538df66b8 00007f952fefb716 00007f952fefb445 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.123553 (+4244500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77421,"reported_tablets":234,"spinlock_wait_cycles":104197376}
I20250110 02:51:33.124073 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.124775 3158 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:33.126201 2289 raft_consensus.cc:491] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.081197 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d4f03a8c66446ea7c6d178cc7ef80a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.126338 2289 raft_consensus.cc:513] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.126646 994 consensus_peers.cc:487] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.126713 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 676f5f457d9e4028bfd0b707604c42f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 467f87955c4a4d27a6f0483aa109b922) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:33.126802 994 leader_election.cc:343] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.126853 2289 leader_election.cc:290] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.126868 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127003 756 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f1d916a5ea48708c2c1a61c92fe137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127045 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.127156 756 tablet_service.cc:1514] Processing DeleteTablet for tablet b43bb790e85343308a1bb1e1110bfb9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127285 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79175853474cbea951f5a5686bf630 with delete_type TABLET_DATA_DELETED (replaced by tablet 08705d260c0a462b94ba6b0dc5121123) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127249 774 raft_consensus.cc:2463] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.083550 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.081843 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e554d00d05b4fa7a14c9b6159f9d726 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.082849 755 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.085949 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.091972 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127560 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.100606 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.101846 749 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.084578 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.102927 2199 raft_consensus.cc:491] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.104175 3105 raft_consensus.cc:491] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.127760 996 leader_election.cc:304] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.127756 2199 raft_consensus.cc:513] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.127784 3105 raft_consensus.cc:513] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.128077 996 leader_election.cc:343] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 818dcbf83303412d97b097913bcd36c3
I20250110 02:51:33.128299 2199 leader_election.cc:290] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.128347 2738 raft_consensus.cc:2798] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.103093 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.128365 3105 leader_election.cc:290] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.128407 2738 raft_consensus.cc:491] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.105106 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9507a5283d8f4d07a293ba474c642298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.105106 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.128461 2738 raft_consensus.cc:3054] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.105815 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6d57f3e9a44d178300d3505d504854 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a9d524fda0d4848b333df9e20a9f68b) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.105849 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.128676 768 raft_consensus.cc:2463] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.106580 754 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9b10c419242cf85ef1b708bd5a862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.109903 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1390f867a3a8436092d1920daff4c5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.110183 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7885a80414f2193400672aa16a911 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.110594 738 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc9a4a842914a6f878d01194cd9a5f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:12 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.110476 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f6df074974974d81a83420e7a3a78070 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.094153 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.107393 740 tablet_service.cc:1514] Processing DeleteTablet for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.127406 756 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.129607 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656aa84a130c42768e969b9322fbd8a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.129673 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f4751d0f64e39b975529c6ac28d8a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.130095 709 leader_election.cc:343] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.130302 709 leader_election.cc:343] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.131142 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656aa84a130c42768e969b9322fbd8a5" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.131242 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f4751d0f64e39b975529c6ac28d8a" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.131413 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818dcbf83303412d97b097913bcd36c3" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.131619 711 leader_election.cc:343] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.131711 711 leader_election.cc:304] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.131915 3105 raft_consensus.cc:2743] T 656aa84a130c42768e969b9322fbd8a5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:33.131920 994 leader_election.cc:343] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.132001 994 leader_election.cc:304] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:33.132062 711 leader_election.cc:343] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.132169 711 leader_election.cc:304] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.132289 2906 raft_consensus.cc:2743] T 818dcbf83303412d97b097913bcd36c3 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.132345 3105 raft_consensus.cc:2743] T 1d2f4751d0f64e39b975529c6ac28d8a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.127230 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 249) took 4020 ms. Trace:
I20250110 02:51:33.132642 996 leader_election.cc:304] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.132582 2357 rpcz_store.cc:276] 0110 02:51:29.106836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.106897 (+ 61us) service_pool.cc:225] Handling call
0110 02:51:33.064182 (+3957285us) ts_tablet_manager.cc:1145] Deleting tablet bb595013e809480a814cb9c7aed856e6
0110 02:51:33.064191 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.068249 (+ 4058us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.079036 (+ 10787us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.127214 (+ 48178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3933736}
I20250110 02:51:33.132987 2357 tablet_replica.cc:331] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.133010 3152 raft_consensus.cc:2798] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.133080 1188 tablet_bootstrap.cc:492] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.133091 3152 raft_consensus.cc:491] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.133157 3152 raft_consensus.cc:3054] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.133173 1188 ts_tablet_manager.cc:1397] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.002s
I20250110 02:51:33.133198 2357 raft_consensus.cc:2238] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.133318 2357 raft_consensus.cc:2267] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.133745 2357 ts_tablet_manager.cc:1905] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.133740 1188 raft_consensus.cc:357] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.133877 1188 raft_consensus.cc:383] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.133917 1188 raft_consensus.cc:738] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.134121 1188 consensus_queue.cc:260] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.134670 1188 ts_tablet_manager.cc:1428] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:33.134864 1188 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.135463 3152 raft_consensus.cc:513] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.135968 3152 leader_election.cc:290] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.136209 1188 tablet_bootstrap.cc:654] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.136920 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:33.137398 994 leader_election.cc:343] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.137900 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]) successfully deleted
I20250110 02:51:33.138962 2352 ts_tablet_manager.cc:1918] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.139458 3105 raft_consensus.cc:491] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.139569 3105 raft_consensus.cc:513] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.139783 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c55788eeddb47aa8a5893703c4d5e67" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.139940 768 raft_consensus.cc:3054] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.139973 1449 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 70e53b838be846c7b3ac44dc75aa3730 consensus state has no leader
I20250110 02:51:33.140199 3105 leader_election.cc:290] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.140956 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6694671f422c4be7b545304a38b17edd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.141299 709 leader_election.cc:343] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.141786 1449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 132) took 2263 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.142160 768 raft_consensus.cc:2463] T 7c55788eeddb47aa8a5893703c4d5e67 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.142942 2352 log.cc:1198] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:33.143262 2352 ts_tablet_manager.cc:1939] T cdb726d2c9a04963b92a3fb625cf84f3 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.143553 2738 raft_consensus.cc:513] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.144001 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 307) took 5422 ms. Trace:
I20250110 02:51:33.144062 2352 rpcz_store.cc:276] 0110 02:51:27.721635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.721669 (+ 34us) service_pool.cc:225] Handling call
0110 02:51:33.097250 (+5375581us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:33.097257 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.108596 (+ 11339us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.138873 (+ 30277us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.143987 (+ 5114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5375466}
I20250110 02:51:33.144163 2738 leader_election.cc:290] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:33.144387 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 308) took 5374 ms. Trace:
I20250110 02:51:33.144424 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet cdb726d2c9a04963b92a3fb625cf84f3 (table default.Table_87bb664f5b24476b9dd295e4fb7b7f20 [id=c24339c8d2fb4da5a888b6de5c22fd95]) successfully deleted
I20250110 02:51:33.145372 3149 tablet.cc:1721] T c1ade70d7fe34776bef3c5853f8dde74 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:33.145782 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.145838 2352 rpcz_store.cc:276] 0110 02:51:27.769615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:27.769663 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:33.144311 (+5374648us) ts_tablet_manager.cc:1145] Deleting tablet 9507a5283d8f4d07a293ba474c642298
0110 02:51:33.144320 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.144372 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5374492}
I20250110 02:51:33.146231 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 311) took 5010 ms. Trace:
I20250110 02:51:33.146332 2352 rpcz_store.cc:276] 0110 02:51:28.135402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.135450 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:33.146131 (+5010681us) ts_tablet_manager.cc:1145] Deleting tablet 33f9b3e74ee2431a85a22fefaacfa40e
0110 02:51:33.146138 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.146218 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5010517}
I20250110 02:51:33.146651 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 310) took 5011 ms. Trace:
I20250110 02:51:33.146749 2352 rpcz_store.cc:276] 0110 02:51:28.135286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.135844 (+ 558us) service_pool.cc:225] Handling call
0110 02:51:33.146575 (+5010731us) ts_tablet_manager.cc:1145] Deleting tablet dc6b2654f86e4686b8a3034de02d11bf
0110 02:51:33.146582 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.146639 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5010566}
I20250110 02:51:33.147595 2352 tablet_replica.cc:331] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.147605 996 leader_election.cc:304] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.147848 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:33.147877 2352 raft_consensus.cc:2238] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.147998 2352 raft_consensus.cc:2267] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.148144 2456 raft_consensus.cc:2798] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.148245 2456 raft_consensus.cc:695] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.148434 2456 consensus_queue.cc:237] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.148155 2908 raft_consensus.cc:491] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.149266 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6694671f422c4be7b545304a38b17edd" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.149394 2908 raft_consensus.cc:513] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.149519 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.149672 768 raft_consensus.cc:3054] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.149950 2908 leader_election.cc:290] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.150931 2352 ts_tablet_manager.cc:1905] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.152420 1181 tablet_bootstrap.cc:492] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.151268 3195 tablet.cc:1721] T c1ade70d7fe34776bef3c5853f8dde74 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:33.152536 1181 ts_tablet_manager.cc:1397] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.007s
I20250110 02:51:33.152550 768 raft_consensus.cc:2463] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.153157 1181 raft_consensus.cc:357] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.153309 1181 raft_consensus.cc:383] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.153362 1181 raft_consensus.cc:738] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.154929 996 leader_election.cc:304] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:33.155669 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.156162 743 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:33.156898 711 leader_election.cc:343] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.157011 711 leader_election.cc:304] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.157215 3105 raft_consensus.cc:2743] T 6694671f422c4be7b545304a38b17edd P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.157533 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9395aa9b39d04ac9b6cecf106bc91499" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.157967 2289 raft_consensus.cc:2798] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.158072 2289 raft_consensus.cc:695] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.158007 853 leader_election.cc:304] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.158249 2908 raft_consensus.cc:2743] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.158243 2289 consensus_queue.cc:237] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:33.149648 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because the tablet was not found. No further retry: Not found: Tablet not found: 9507a5283d8f4d07a293ba474c642298
W20250110 02:51:33.160898 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 33f9b3e74ee2431a85a22fefaacfa40e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f9b3e74ee2431a85a22fefaacfa40e already in progress: creating tablet
W20250110 02:51:33.161123 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet dc6b2654f86e4686b8a3034de02d11bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc6b2654f86e4686b8a3034de02d11bf already in progress: creating tablet
W20250110 02:51:33.141889 1449 rpcz_store.cc:269] 0110 02:51:30.878676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.198702 (+320026us) service_pool.cc:225] Handling call
0110 02:51:32.395857 (+1197155us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.395865 (+ 8us) write_op.cc:454] Released schema lock
0110 02:51:33.141738 (+745873us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:32.385413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:32.385541 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:32.385549 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:32.385552 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:32.385662 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:32.385678 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:32.385685 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:32.385754 (+ 69us) tablet.cc:698] Row locks acquired
0110 02:51:32.385757 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:32.385793 (+ 36us) write_op.cc:270] Start()
0110 02:51:32.385826 (+ 33us) write_op.cc:276] Timestamp: P: 1736477492385738 usec, L: 0
0110 02:51:32.385830 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:32.386064 (+ 234us) log.cc:844] Serialized 527 byte log entry
0110 02:51:32.392819 (+ 6755us) op_driver.cc:464] REPLICATION: finished
0110 02:51:32.392988 (+ 169us) write_op.cc:301] APPLY: starting
0110 02:51:32.393007 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:32.393057 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:32.393100 (+ 43us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:32.393156 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:32.393190 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:32.393198 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:32.393276 (+ 78us) log.cc:844] Serialized 23 byte log entry
0110 02:51:32.393338 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.393358 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:32.393365 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":65671,"reported_tablets":230,"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":435,"num_ops":1,"prepare.queue_time_us":2805,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":753,"raft.queue_time_us":2093,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":223,"replication_time_us":6952}]]}
I20250110 02:51:33.161765 1449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_98b0cdca19ab460ca64c35faf958d9bb" } modify_external_catalogs: true
I20250110 02:51:33.161973 1449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_98b0cdca19ab460ca64c35faf958d9bb" } modify_external_catalogs: true
I20250110 02:51:33.163301 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1ade70d7fe34776bef3c5853f8dde74 (table default.Table_83c9397ae7e84b17a964ab1d664307b2 [id=88412101305b4d0a8dea71c08ff391aa]). Expected version 1 got 0
I20250110 02:51:33.163887 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:33.164625 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:33.167043 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:33.167265 994 consensus_peers.cc:487] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:33.167536 994 leader_election.cc:343] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.168087 3150 tablet.cc:1721] T c1ade70d7fe34776bef3c5853f8dde74 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 02:51:33.169384 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:33.169759 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 1 got 0
I20250110 02:51:33.170089 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:33.170367 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:33.170696 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:33.171072 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:33.171397 1439 catalog_manager.cc:5618] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:33.172629 2456 consensus_queue.cc:1035] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.175571 2352 ts_tablet_manager.cc:1918] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.176899 3105 raft_consensus.cc:491] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.177009 3105 raft_consensus.cc:513] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.177497 3105 leader_election.cc:290] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.179104 2908 raft_consensus.cc:491] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.179638 2908 raft_consensus.cc:513] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.179847 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d9140c04a14afc96d43f3bc93488e9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.180192 2908 leader_election.cc:290] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:33.180238 709 leader_election.cc:343] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.180416 2352 log.cc:1198] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f69d0c201c524b7e95c9d3851b8dd8c2
I20250110 02:51:33.180883 2352 ts_tablet_manager.cc:1939] T f69d0c201c524b7e95c9d3851b8dd8c2 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.180817 1181 consensus_queue.cc:260] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.181782 1181 ts_tablet_manager.cc:1428] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250110 02:51:33.182345 1181 tablet_bootstrap.cc:492] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.183924 1184 tablet_bootstrap.cc:654] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.185365 1181 tablet_bootstrap.cc:654] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.191272 2357 ts_tablet_manager.cc:1918] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.194711 1031 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.194778 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.195006 2357 log.cc:1198] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/f6636a3b7f2c4e2fb0ef76f9ab25dcfb
I20250110 02:51:33.195092 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b90cc15dbf47d4b4dbaf12d588abb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.195359 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.195492 2357 ts_tablet_manager.cc:1939] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.195647 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.196566 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 250) took 4065 ms. Trace:
I20250110 02:51:33.196570 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d9140c04a14afc96d43f3bc93488e9" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.196734 2357 rpcz_store.cc:276] 0110 02:51:29.131248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.131296 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:33.132944 (+4001648us) ts_tablet_manager.cc:1145] Deleting tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb
0110 02:51:33.132954 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.186043 (+ 53089us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.191189 (+ 5146us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.196551 (+ 5362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4001494}
I20250110 02:51:33.197351 2357 tablet_replica.cc:331] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.197549 1444 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_f752952b67ed41d48e3e639949d6869a [id=f6b39e5377964d51adc9632b8199e4f5] alter complete (version 2)
I20250110 02:51:33.197661 2357 raft_consensus.cc:2238] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.197850 2357 raft_consensus.cc:2267] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.198963 2357 ts_tablet_manager.cc:1905] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.198966 1448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_8de67d45a9a54567afbcd1896b321c97 [id=38abb97bc5e34b318c9a9081c1cdc7bd] alter complete (version 1)
I20250110 02:51:33.201893 1456 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_bc967f7066e24bb9b02c668d1f2b2be2 [id=8c652eff76844e768fb79ff4bfa9780f] alter complete (version 1)
I20250110 02:51:33.209573 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]) successfully deleted
W20250110 02:51:33.211051 711 leader_election.cc:343] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.211434 711 leader_election.cc:304] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.212081 3105 raft_consensus.cc:2743] T 78d9140c04a14afc96d43f3bc93488e9 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:33.212183 1440 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 113) took 2354 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.212728 1184 tablet_bootstrap.cc:492] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.212893 1184 ts_tablet_manager.cc:1397] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.239s user 0.007s sys 0.005s
W20250110 02:51:33.213132 1440 rpcz_store.cc:269] 0110 02:51:30.857350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.187819 (+330469us) service_pool.cc:225] Handling call
0110 02:51:31.461441 (+273622us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.461448 (+ 7us) write_op.cc:454] Released schema lock
0110 02:51:33.212139 (+1750691us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.453270 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.453394 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.453402 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.453407 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:31.453591 (+ 184us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.453626 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.453633 (+ 7us) tablet.cc:682] Acquiring locks for 3 operations
0110 02:51:31.453752 (+ 119us) tablet.cc:698] Row locks acquired
0110 02:51:31.453756 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.453805 (+ 49us) write_op.cc:270] Start()
0110 02:51:31.453842 (+ 37us) write_op.cc:276] Timestamp: P: 1736477491453739 usec, L: 0
0110 02:51:31.453847 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.454036 (+ 189us) log.cc:844] Serialized 1275 byte log entry
0110 02:51:31.457740 (+ 3704us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.458139 (+ 399us) write_op.cc:301] APPLY: starting
0110 02:51:31.458155 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.458213 (+ 58us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.458242 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.458327 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.458390 (+ 63us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0110 02:51:31.458400 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:31.458511 (+ 111us) log.cc:844] Serialized 35 byte log entry
0110 02:51:31.458576 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.458600 (+ 24us) write_op.cc:454] Released schema lock
0110 02:51:31.458609 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":332,"apply.run_cpu_time_us":542,"apply.run_wall_time_us":541,"num_ops":3,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":869,"prepare.run_wall_time_us":869,"raft.queue_time_us":122,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":240,"replication_time_us":3861}]]}
I20250110 02:51:33.213549 1184 raft_consensus.cc:357] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.213950 1184 raft_consensus.cc:383] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.214051 1184 raft_consensus.cc:738] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.214293 1184 consensus_queue.cc:260] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.214936 1184 ts_tablet_manager.cc:1428] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.215250 1184 tablet_bootstrap.cc:492] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.217418 1184 tablet_bootstrap.cc:654] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:33.196743 853 leader_election.cc:343] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:33.205188 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 with delete_type TABLET_DATA_DELETED (replaced by tablet deaa3527aa074e6ba94960c672aee028) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.212646 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.212782 896 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.212925 898 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.213032 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.213140 886 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa with delete_type TABLET_DATA_DELETED (replaced by tablet 248055ca83de4e62a6b7e908c53d3d9a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.213351 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.213438 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:33.217916 853 leader_election.cc:343] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 19b90cc15dbf47d4b4dbaf12d588abb0
I20250110 02:51:33.213899 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214032 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.218014 853 leader_election.cc:304] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.214125 894 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214211 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e015a41a03f84c91a45e46e4b7b5e7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214293 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d761ef0d1e5542b1979c29bb2d579489 with delete_type TABLET_DATA_DELETED (replaced by tablet 071bbce1706c43b6bb690c7de1ead3fe) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214367 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:16 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.218390 2908 raft_consensus.cc:2743] T 19b90cc15dbf47d4b4dbaf12d588abb0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.214468 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 302af84fd8374ffa96a7503b54849536 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214545 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 3510e86314144a2ca012aacf3d78eacc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.214643 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f15078fc22f4b32a48c7190b36e2192 with delete_type TABLET_DATA_DELETED (replaced by tablet bfd120d8edf14151b7ebed3b74e2b399) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.218946 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.219286 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.219564 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.219841 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.220137 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.220494 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 312) took 5085 ms. Trace:
I20250110 02:51:33.220626 2352 rpcz_store.cc:276] 0110 02:51:28.135476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.136070 (+ 594us) service_pool.cc:225] Handling call
0110 02:51:33.147561 (+5011491us) ts_tablet_manager.cc:1145] Deleting tablet f69d0c201c524b7e95c9d3851b8dd8c2
0110 02:51:33.147568 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.167688 (+ 20120us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.175504 (+ 7816us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.220477 (+ 44973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5011355}
I20250110 02:51:33.220968 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.221138 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f69d0c201c524b7e95c9d3851b8dd8c2 (table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]) successfully deleted
I20250110 02:51:33.221129 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 313) took 5085 ms. Trace:
I20250110 02:51:33.221230 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.221450 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.221560 2357 ts_tablet_manager.cc:1918] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.221653 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d69d04b494a848653636c9e259105 with delete_type TABLET_DATA_DELETED (replaced by tablet 24034fb3da6e47978cdc9b02e714e30d) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.221560 2352 rpcz_store.cc:276] 0110 02:51:28.135609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.136235 (+ 626us) service_pool.cc:225] Handling call
0110 02:51:33.220956 (+5084721us) ts_tablet_manager.cc:1145] Deleting tablet d6acae633eb14e5db87b1e36a069422c
0110 02:51:33.220968 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.221111 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5084600}
I20250110 02:51:33.221874 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 314) took 5086 ms. Trace:
I20250110 02:51:33.221894 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 971f97a3be9b49958b1218a275e126d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bff349588d44a7dae691bc1d2e2886a) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:33.221920 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d6acae633eb14e5db87b1e36a069422c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6acae633eb14e5db87b1e36a069422c already in progress: creating tablet
I20250110 02:51:33.222167 2289 raft_consensus.cc:491] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.222275 2289 raft_consensus.cc:513] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.222856 2289 leader_election.cc:290] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:33.223435 996 leader_election.cc:343] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: c28f191afa2a43608bbd74322bec97a6
W20250110 02:51:33.223457 1450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 123) took 3717 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.223601 2357 log.cc:1198] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/3ca895eca14748c19fdb7d363499b544
W20250110 02:51:33.223551 1450 rpcz_store.cc:269] 0110 02:51:29.505996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:30.410127 (+904131us) service_pool.cc:225] Handling call
0110 02:51:33.223427 (+2813300us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.228485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.228619 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.228627 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:31.228630 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:31.228808 (+ 178us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.228822 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.228827 (+ 5us) tablet.cc:682] Acquiring locks for 5 operations
0110 02:51:31.229022 (+ 195us) tablet.cc:698] Row locks acquired
0110 02:51:31.229024 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:31.229058 (+ 34us) write_op.cc:270] Start()
0110 02:51:31.229091 (+ 33us) write_op.cc:276] Timestamp: P: 1736477491229015 usec, L: 0
0110 02:51:31.229095 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.229274 (+ 179us) log.cc:844] Serialized 2024 byte log entry
0110 02:51:31.235735 (+ 6461us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.235838 (+ 103us) write_op.cc:301] APPLY: starting
0110 02:51:31.235853 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.237663 (+ 1810us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.237690 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.237799 (+ 109us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.237841 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0110 02:51:31.237852 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:31.237938 (+ 86us) log.cc:844] Serialized 47 byte log entry
0110 02:51:31.238021 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.238051 (+ 30us) write_op.cc:454] Released schema lock
0110 02:51:31.238060 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:31.238149 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.238155 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":145032,"reported_tablets":248,"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":646,"apply.run_wall_time_us":2288,"num_ops":5,"prepare.queue_time_us":402,"prepare.run_cpu_time_us":886,"prepare.run_wall_time_us":887,"raft.queue_time_us":322,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":901,"replication_time_us":6613}]]}
I20250110 02:51:33.224525 2357 ts_tablet_manager.cc:1939] T 3ca895eca14748c19fdb7d363499b544 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.223306 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28f191afa2a43608bbd74322bec97a6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.224675 2352 rpcz_store.cc:276] 0110 02:51:28.135704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.137910 (+ 2206us) service_pool.cc:225] Handling call
0110 02:51:33.221787 (+5083877us) ts_tablet_manager.cc:1145] Deleting tablet e848586bec574a529caaf5e498db79cd
0110 02:51:33.221795 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.221862 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5082369}
W20250110 02:51:33.224972 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e848586bec574a529caaf5e498db79cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e848586bec574a529caaf5e498db79cd already in progress: creating tablet
W20250110 02:51:33.225049 994 leader_election.cc:343] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.225138 994 leader_election.cc:304] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.225239 2352 tablet_replica.cc:331] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:33.225308 1457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 124) took 2446 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.225419 2289 raft_consensus.cc:2743] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.226014 1181 tablet_bootstrap.cc:492] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
W20250110 02:51:33.225401 1457 rpcz_store.cc:269] 0110 02:51:30.779105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.181365 (+402260us) service_pool.cc:225] Handling call
0110 02:51:31.796268 (+614903us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.796273 (+ 5us) write_op.cc:454] Released schema lock
0110 02:51:32.100217 (+303944us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.100234 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:33.225280 (+1125046us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:31.791453 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:31.791547 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:31.791552 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:31.791554 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:31.791741 (+ 187us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:31.791752 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:31.791756 (+ 4us) tablet.cc:682] Acquiring locks for 12 operations
0110 02:51:31.791999 (+ 243us) tablet.cc:698] Row locks acquired
0110 02:51:31.792003 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:31.792036 (+ 33us) write_op.cc:270] Start()
0110 02:51:31.792070 (+ 34us) write_op.cc:276] Timestamp: P: 1736477491791989 usec, L: 0
0110 02:51:31.792075 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:31.792252 (+ 177us) log.cc:844] Serialized 4642 byte log entry
0110 02:51:31.795326 (+ 3074us) op_driver.cc:464] REPLICATION: finished
0110 02:51:31.795422 (+ 96us) write_op.cc:301] APPLY: starting
0110 02:51:31.795434 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:31.795544 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:31.795591 (+ 47us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:31.795732 (+ 141us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:31.795763 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0110 02:51:31.795770 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:31.795843 (+ 73us) log.cc:844] Serialized 89 byte log entry
0110 02:51:31.795892 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:31.795917 (+ 25us) write_op.cc:454] Released schema lock
0110 02:51:31.795923 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:32.093825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:32.093956 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:32.093963 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:32.093965 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:32.094073 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:32.094087 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:32.094094 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:32.094168 (+ 74us) tablet.cc:698] Row locks acquired
0110 02:51:32.094171 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:32.094202 (+ 31us) write_op.cc:270] Start()
0110 02:51:32.094233 (+ 31us) write_op.cc:276] Timestamp: P: 1736477492094178 usec, L: 0
0110 02:51:32.094236 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:32.094424 (+ 188us) log.cc:844] Serialized 356 byte log entry
0110 02:51:32.097452 (+ 3028us) op_driver.cc:464] REPLICATION: finished
0110 02:51:32.097552 (+ 100us) write_op.cc:301] APPLY: starting
0110 02:51:32.097566 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:32.097604 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:32.097628 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:32.097676 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:32.097710 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:32.097716 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:32.097789 (+ 73us) log.cc:844] Serialized 23 byte log entry
0110 02:51:32.097841 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.097857 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:32.097863 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":101626,"reported_tablets":240,"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":544,"apply.run_wall_time_us":544,"num_ops":12,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":884,"prepare.run_wall_time_us":884,"raft.queue_time_us":468,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":289,"replication_time_us":3227}],["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":367,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":686,"raft.queue_time_us":368,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":3193}]]}
I20250110 02:51:33.226521 1181 ts_tablet_manager.cc:1397] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.004s
I20250110 02:51:33.227111 1181 raft_consensus.cc:357] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.227316 1181 raft_consensus.cc:383] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.227394 1181 raft_consensus.cc:738] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.227596 1181 consensus_queue.cc:260] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.227869 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dcbf83303412d97b097913bcd36c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.228286 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f229f546a8d4a09b93f8b46c893d14a with delete_type TABLET_DATA_DELETED (replaced by tablet dc6b2654f86e4686b8a3034de02d11bf) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.225396 2352 raft_consensus.cc:2238] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.229317 1181 ts_tablet_manager.cc:1428] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:33.229422 2352 raft_consensus.cc:2267] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:33.230499 1441 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:37843 (request call id 122) took 4807 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.230551 2352 ts_tablet_manager.cc:1905] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.231097 1181 tablet_bootstrap.cc:492] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.230756 911 raft_consensus.cc:3054] T d3858638b6fc48d0bb8b8f97370a791d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.248315 1181 tablet_bootstrap.cc:654] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.249635 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 257) took 2834 ms. Trace:
I20250110 02:51:33.250645 2352 ts_tablet_manager.cc:1918] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.253837 2357 rpcz_store.cc:276] 0110 02:51:30.415394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:30.415449 (+ 55us) service_pool.cc:225] Handling call
0110 02:51:33.197296 (+2781847us) ts_tablet_manager.cc:1145] Deleting tablet 3ca895eca14748c19fdb7d363499b544
0110 02:51:33.197309 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.201614 (+ 4305us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.221468 (+ 19854us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.249615 (+ 28147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2781693}
I20250110 02:51:33.254176 2357 tablet_replica.cc:331] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.254357 2357 raft_consensus.cc:2238] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.254523 2357 raft_consensus.cc:2267] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.254973 2357 ts_tablet_manager.cc:1905] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.260548 2224 ts_tablet_manager.cc:1918] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.255842 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 3ca895eca14748c19fdb7d363499b544 (table default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]) successfully deleted
I20250110 02:51:33.265089 2908 raft_consensus.cc:491] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.265260 914 raft_consensus.cc:3054] T b010e9c549214937889fa281542e752e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.265584 2908 raft_consensus.cc:513] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.266139 2908 leader_election.cc:290] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:33.264636 1441 rpcz_store.cc:269] 0110 02:51:28.423126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.479714 (+ 56588us) service_pool.cc:225] Handling call
0110 02:51:28.679140 (+199426us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538df66b8 00007f952fefb716 00007f952fefb445 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.230453 (+4551313us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:30.081129 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:30.081252 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:30.081260 (+ 8us) write_op.cc:435] Acquired schema lock
0110 02:51:30.081264 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:30.081663 (+ 399us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:30.081676 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:30.081680 (+ 4us) tablet.cc:682] Acquiring locks for 4 operations
0110 02:51:30.081778 (+ 98us) tablet.cc:698] Row locks acquired
0110 02:51:30.081780 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:30.081816 (+ 36us) write_op.cc:270] Start()
0110 02:51:30.081854 (+ 38us) write_op.cc:276] Timestamp: P: 1736477490081793 usec, L: 0
0110 02:51:30.081857 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:30.082098 (+ 241us) log.cc:844] Serialized 1650 byte log entry
0110 02:51:30.171511 (+ 89413us) op_driver.cc:464] REPLICATION: finished
0110 02:51:30.177594 (+ 6083us) write_op.cc:301] APPLY: starting
0110 02:51:30.177618 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:30.177707 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:30.177736 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:30.181663 (+ 3927us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:30.181706 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0110 02:51:30.181715 (+ 9us) write_op.cc:312] APPLY: finished
0110 02:51:30.181835 (+ 120us) log.cc:844] Serialized 41 byte log entry
0110 02:51:30.182136 (+ 301us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.182162 (+ 26us) write_op.cc:454] Released schema lock
0110 02:51:30.182170 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:30.182257 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:30.182262 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1979,"reported_tablets":250,"spinlock_wait_cycles":243920640,"child_traces":[["op",{"apply.queue_time_us":5990,"apply.run_cpu_time_us":705,"apply.run_wall_time_us":4666,"num_ops":4,"prepare.queue_time_us":9495,"prepare.run_cpu_time_us":1103,"prepare.run_wall_time_us":11219,"raft.queue_time_us":32,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":230,"replication_time_us":89608,"thread_start_us":16289,"threads_started":2}]]}
I20250110 02:51:33.269614 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.271778 2352 log.cc:1198] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:33.272325 2352 ts_tablet_manager.cc:1939] T 4de67338a26f4febb5dd7611a72b4723 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.277900 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 315) took 5059 ms. Trace:
I20250110 02:51:33.279739 2352 rpcz_store.cc:276] 0110 02:51:28.218079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.218134 (+ 55us) service_pool.cc:225] Handling call
0110 02:51:33.225205 (+5007071us) ts_tablet_manager.cc:1145] Deleting tablet 4de67338a26f4febb5dd7611a72b4723
0110 02:51:33.225213 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.240783 (+ 15570us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.250558 (+ 9775us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.277882 (+ 27324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5006888}
I20250110 02:51:33.280826 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 4de67338a26f4febb5dd7611a72b4723 (table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]) successfully deleted
I20250110 02:51:33.273370 911 raft_consensus.cc:3054] T b25bbf8aa71e478587093627c715310f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.281150 2352 tablet_replica.cc:331] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.281618 2352 raft_consensus.cc:2238] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.282286 2352 raft_consensus.cc:2267] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.283922 1450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_1a66990aee93403b8d03464dec9acd0d" } modify_external_catalogs: true
I20250110 02:51:33.284117 1450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_1a66990aee93403b8d03464dec9acd0d" } modify_external_catalogs: true
I20250110 02:51:33.285045 2352 ts_tablet_manager.cc:1905] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.285406 1457 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_94cd6ab467f442cba51ee627721ab3e8" } modify_external_catalogs: true
I20250110 02:51:33.285591 1457 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_94cd6ab467f442cba51ee627721ab3e8" } modify_external_catalogs: true
I20250110 02:51:33.286633 1440 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_8de67d45a9a54567afbcd1896b321c97" } new_table_name: "default.Table_9b156f13a71144d188c338b19ebe5f7e" modify_external_catalogs: true
I20250110 02:51:33.287720 1441 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_7cd682c4f72745b5aa221478659d3350" } new_table_name: "default.Table_95e38b56742441e987008a40d45750fc" modify_external_catalogs: true
I20250110 02:51:33.297261 3105 raft_consensus.cc:491] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.297386 3105 raft_consensus.cc:513] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.297904 3105 leader_election.cc:290] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.298599 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a254ed6c239249c6982429eb5e142031" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.298988 2352 ts_tablet_manager.cc:1918] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 02:51:33.299212 709 leader_election.cc:343] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.264432 2289 raft_consensus.cc:491] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.299409 2289 raft_consensus.cc:513] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.299669 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a254ed6c239249c6982429eb5e142031" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.300000 2289 leader_election.cc:290] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.271585 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3033595f824affb2d300c3964e1e8e" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:33.300072 711 leader_election.cc:343] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.300149 711 leader_election.cc:304] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.300731 2224 log.cc:1198] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/33f9b3e74ee2431a85a22fefaacfa40e
I20250110 02:51:33.300729 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.300920 772 raft_consensus.cc:2463] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.301128 3105 raft_consensus.cc:2743] T a254ed6c239249c6982429eb5e142031 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.301349 2224 ts_tablet_manager.cc:1939] T 33f9b3e74ee2431a85a22fefaacfa40e P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.301546 996 leader_election.cc:304] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:33.301782 2289 raft_consensus.cc:2798] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.301849 2289 raft_consensus.cc:491] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.301913 2289 raft_consensus.cc:3054] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.304344 2289 raft_consensus.cc:513] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.304965 2289 leader_election.cc:290] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.300585 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.302043 853 leader_election.cc:304] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.305593 2908 raft_consensus.cc:2743] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.305214 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:33.305989 2352 log.cc:1198] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7c2c7a313ba94e5ea85e45276658ba04
W20250110 02:51:33.306231 994 leader_election.cc:343] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.306473 994 leader_election.cc:343] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.306892 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf81039d5e4a47b426ea5d2f21acc2" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.306691 1188 tablet_bootstrap.cc:492] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.307082 772 raft_consensus.cc:3054] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.307245 1188 ts_tablet_manager.cc:1397] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.172s user 0.009s sys 0.018s
I20250110 02:51:33.308157 1188 raft_consensus.cc:357] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.308425 1188 raft_consensus.cc:383] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.308539 1188 raft_consensus.cc:738] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.308825 1188 consensus_queue.cc:260] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.309360 772 raft_consensus.cc:2463] T 24cf81039d5e4a47b426ea5d2f21acc2 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.309502 1188 ts_tablet_manager.cc:1428] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:33.309780 1188 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.310159 996 leader_election.cc:304] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.310923 2456 raft_consensus.cc:2798] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.307008 2352 ts_tablet_manager.cc:1939] T 7c2c7a313ba94e5ea85e45276658ba04 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.311041 2456 raft_consensus.cc:695] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.311242 2456 consensus_queue.cc:237] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.311842 1188 tablet_bootstrap.cc:654] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.312278 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 316) took 5044 ms. Trace:
I20250110 02:51:33.313097 2352 rpcz_store.cc:276] 0110 02:51:28.267820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.267871 (+ 51us) service_pool.cc:225] Handling call
0110 02:51:33.281102 (+5013231us) ts_tablet_manager.cc:1145] Deleting tablet 7c2c7a313ba94e5ea85e45276658ba04
0110 02:51:33.281113 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.296383 (+ 15270us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.298903 (+ 2520us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.312261 (+ 13358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5012857}
I20250110 02:51:33.313594 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 317) took 4973 ms. Trace:
I20250110 02:51:33.313551 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7c2c7a313ba94e5ea85e45276658ba04 (table default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]) successfully deleted
I20250110 02:51:33.313710 2352 rpcz_store.cc:276] 0110 02:51:28.339900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.339950 (+ 50us) service_pool.cc:225] Handling call
0110 02:51:33.313501 (+4973551us) ts_tablet_manager.cc:1145] Deleting tablet 9b937188bf2949af8f54bade4e0ad984
0110 02:51:33.313509 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.313581 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4973389}
W20250110 02:51:33.314100 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9b937188bf2949af8f54bade4e0ad984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b937188bf2949af8f54bade4e0ad984 already in progress: creating tablet
I20250110 02:51:33.314131 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 319) took 4956 ms. Trace:
I20250110 02:51:33.314757 1181 tablet_bootstrap.cc:492] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.314877 1181 ts_tablet_manager.cc:1397] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20250110 02:51:33.314869 2352 rpcz_store.cc:276] 0110 02:51:28.357365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.357406 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:33.314066 (+4956660us) ts_tablet_manager.cc:1145] Deleting tablet e744cdc1fcb94d46b48b684763ecf1bf
0110 02:51:33.314074 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.314119 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4956428}
W20250110 02:51:33.315315 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e744cdc1fcb94d46b48b684763ecf1bf because the tablet was not found. No further retry: Not found: Tablet not found: e744cdc1fcb94d46b48b684763ecf1bf
I20250110 02:51:33.315536 1181 raft_consensus.cc:357] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.315739 1181 raft_consensus.cc:383] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.315819 1181 raft_consensus.cc:738] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.316044 1181 consensus_queue.cc:260] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.316601 1181 ts_tablet_manager.cc:1428] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:33.317025 1184 tablet_bootstrap.cc:492] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.317191 1184 ts_tablet_manager.cc:1397] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.012s
I20250110 02:51:33.318367 2352 tablet_replica.cc:331] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.317829 1184 raft_consensus.cc:357] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.318491 1184 raft_consensus.cc:383] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.318554 1184 raft_consensus.cc:738] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.318732 2352 raft_consensus.cc:2238] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.318745 1184 consensus_queue.cc:260] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.319411 1184 ts_tablet_manager.cc:1428] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.319458 1181 tablet_bootstrap.cc:492] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.319749 2352 raft_consensus.cc:2267] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.320243 2352 ts_tablet_manager.cc:1905] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.322590 1184 tablet_bootstrap.cc:492] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.323958 1184 tablet_bootstrap.cc:654] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.325510 1181 tablet_bootstrap.cc:654] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.326838 911 raft_consensus.cc:3054] T ae3033595f824affb2d300c3964e1e8e P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.329254 2357 ts_tablet_manager.cc:1918] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.330152 2352 ts_tablet_manager.cc:1918] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.331149 1188 tablet_bootstrap.cc:492] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.331295 1188 ts_tablet_manager.cc:1397] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250110 02:51:33.331324 2357 log.cc:1198] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/0558a2c03f6b42a6befdc48e3af23a6d
I20250110 02:51:33.331625 2357 ts_tablet_manager.cc:1939] T 0558a2c03f6b42a6befdc48e3af23a6d P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.331948 1188 raft_consensus.cc:357] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.332096 1188 raft_consensus.cc:383] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.332144 1188 raft_consensus.cc:738] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.332281 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 273) took 2139 ms. Trace:
I20250110 02:51:33.332345 2357 rpcz_store.cc:276] 0110 02:51:31.192905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.193499 (+ 594us) service_pool.cc:225] Handling call
0110 02:51:33.254139 (+2060640us) ts_tablet_manager.cc:1145] Deleting tablet 0558a2c03f6b42a6befdc48e3af23a6d
0110 02:51:33.254147 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.308571 (+ 54424us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.329171 (+ 20600us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.332263 (+ 3092us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2060449}
I20250110 02:51:33.332613 2357 tablet_replica.cc:331] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.335907 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 0558a2c03f6b42a6befdc48e3af23a6d (table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]) successfully deleted
I20250110 02:51:33.335961 2908 raft_consensus.cc:491] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.336061 2908 raft_consensus.cc:513] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.336272 1188 consensus_queue.cc:260] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.336609 2908 leader_election.cc:290] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.336722 1188 ts_tablet_manager.cc:1428] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:33.337112 1188 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.337190 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.337380 772 raft_consensus.cc:2388] T 2b2b5e3c14244418912f344b7519995d P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:33.337546 2357 raft_consensus.cc:2238] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.338017 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b5e3c14244418912f344b7519995d" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.338253 1184 tablet_bootstrap.cc:492] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.338353 1184 ts_tablet_manager.cc:1397] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250110 02:51:33.338485 853 leader_election.cc:304] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.338855 2908 raft_consensus.cc:3054] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.338968 1188 tablet_bootstrap.cc:654] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.339316 1184 raft_consensus.cc:357] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.339542 1184 raft_consensus.cc:383] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.339643 1184 raft_consensus.cc:738] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.339905 1184 consensus_queue.cc:260] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.340452 1184 ts_tablet_manager.cc:1428] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.340476 2352 log.cc:1198] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:33.340808 2352 ts_tablet_manager.cc:1939] T c00e97e496b54dcf82b7182591d6f4aa P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.340823 1184 tablet_bootstrap.cc:492] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.341274 2908 raft_consensus.cc:2743] T 2b2b5e3c14244418912f344b7519995d P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.341964 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 320) took 4984 ms. Trace:
I20250110 02:51:33.342042 2352 rpcz_store.cc:276] 0110 02:51:28.357446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.357670 (+ 224us) service_pool.cc:225] Handling call
0110 02:51:33.318328 (+4960658us) ts_tablet_manager.cc:1145] Deleting tablet c00e97e496b54dcf82b7182591d6f4aa
0110 02:51:33.318336 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.327460 (+ 9124us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.330084 (+ 2624us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.341948 (+ 11864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4960482}
I20250110 02:51:33.342432 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet c00e97e496b54dcf82b7182591d6f4aa (table default.Table_f051d394f83d4eb39d7d6a458a478718 [id=d2a1f0bbdcf84021a1aaa9082b64b07c]) successfully deleted
I20250110 02:51:33.342864 2352 tablet_replica.cc:331] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.343628 2352 raft_consensus.cc:2238] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.343909 2352 raft_consensus.cc:2267] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.344362 2352 ts_tablet_manager.cc:1905] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.345325 1184 tablet_bootstrap.cc:654] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.337731 2357 raft_consensus.cc:2267] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.325927 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 346) took 5180 ms. Trace:
I20250110 02:51:33.346705 2357 ts_tablet_manager.cc:1905] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.346819 2224 rpcz_store.cc:276] 0110 02:51:28.145416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.145577 (+ 161us) service_pool.cc:225] Handling call
0110 02:51:32.948209 (+4802632us) ts_tablet_manager.cc:1145] Deleting tablet 33f9b3e74ee2431a85a22fefaacfa40e
0110 02:51:32.948218 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.103095 (+154877us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.260442 (+157347us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.325909 (+ 65467us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47339,"tablet-delete.queue_time_us":4802428}
I20250110 02:51:33.347380 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 345) took 5202 ms. Trace:
I20250110 02:51:33.349992 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 33f9b3e74ee2431a85a22fefaacfa40e (table default.Table_a2d5d1bf3c504747b630f2dd924d83f7 [id=3ee56bd0749a45718767531e0dba1f92]) successfully deleted
I20250110 02:51:33.350252 2224 rpcz_store.cc:276] 0110 02:51:28.144985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.146062 (+ 1077us) service_pool.cc:225] Handling call
0110 02:51:33.347274 (+5201212us) ts_tablet_manager.cc:1145] Deleting tablet dc6b2654f86e4686b8a3034de02d11bf
0110 02:51:33.347283 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.347366 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5201067}
W20250110 02:51:33.350653 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet dc6b2654f86e4686b8a3034de02d11bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc6b2654f86e4686b8a3034de02d11bf already in progress: creating tablet
W20250110 02:51:33.351119 994 consensus_peers.cc:487] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.351819 2224 tablet_replica.cc:331] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.352051 2224 raft_consensus.cc:2238] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.355125 2224 raft_consensus.cc:2267] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.355656 2224 ts_tablet_manager.cc:1905] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.357331 1184 tablet_bootstrap.cc:492] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.357497 1184 ts_tablet_manager.cc:1397] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250110 02:51:33.357672 2352 ts_tablet_manager.cc:1918] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.357766 2352 log.cc:1198] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:33.358058 2352 ts_tablet_manager.cc:1939] T 80186dd7d1a240068584b64ce9e9b03f P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.358443 1184 raft_consensus.cc:357] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.358786 1184 raft_consensus.cc:383] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.358906 1184 raft_consensus.cc:738] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.359247 1184 consensus_queue.cc:260] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.360199 1184 ts_tablet_manager.cc:1428] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250110 02:51:33.362406 911 raft_consensus.cc:3054] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.365510 1184 tablet_bootstrap.cc:492] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.368405 2199 raft_consensus.cc:491] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.368511 2199 raft_consensus.cc:513] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.369086 2199 leader_election.cc:290] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.369434 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5134444da58427aa97e166e44a3a648" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.369632 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5134444da58427aa97e166e44a3a648" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.370076 711 leader_election.cc:343] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.370175 709 leader_election.cc:343] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.370281 709 leader_election.cc:304] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.370517 914 raft_consensus.cc:3054] T 04a4e7142e83493b98f71487df3f304d P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.370522 3105 raft_consensus.cc:491] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.370607 3105 raft_consensus.cc:513] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.371109 3105 leader_election.cc:290] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.371219 3105 raft_consensus.cc:2743] T b5134444da58427aa97e166e44a3a648 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.371851 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.372342 709 leader_election.cc:343] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.372400 2357 ts_tablet_manager.cc:1918] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.372470 2357 log.cc:1198] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/eed058931194480ca1a27130bfa8ee32
I20250110 02:51:33.373205 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.373425 1057 raft_consensus.cc:2463] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:33.372661 1184 tablet_bootstrap.cc:654] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.373497 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 321) took 5015 ms. Trace:
I20250110 02:51:33.373574 2352 rpcz_store.cc:276] 0110 02:51:28.357541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.357885 (+ 344us) service_pool.cc:225] Handling call
0110 02:51:33.342824 (+4984939us) ts_tablet_manager.cc:1145] Deleting tablet 80186dd7d1a240068584b64ce9e9b03f
0110 02:51:33.342833 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.355075 (+ 12242us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.357596 (+ 2521us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.373479 (+ 15883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4984772}
I20250110 02:51:33.373899 711 leader_election.cc:304] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.374039 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 322) took 5016 ms. Trace:
I20250110 02:51:33.374164 2199 raft_consensus.cc:2798] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.374243 2199 raft_consensus.cc:491] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.374305 2199 raft_consensus.cc:3054] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.374585 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 80186dd7d1a240068584b64ce9e9b03f (table default.Table_f40ec2d4223143dcb1175f357f06de68 [id=f9c9a5ff528a425d8bf25fdc2bef2931]) successfully deleted
I20250110 02:51:33.375479 2357 ts_tablet_manager.cc:1939] T eed058931194480ca1a27130bfa8ee32 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.376493 2199 raft_consensus.cc:513] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.377012 2199 leader_election.cc:290] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.377558 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:33.377759 1057 raft_consensus.cc:3054] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.378365 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 274) took 2174 ms. Trace:
I20250110 02:51:33.378458 2357 rpcz_store.cc:276] 0110 02:51:31.203988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.204810 (+ 822us) service_pool.cc:225] Handling call
0110 02:51:33.332574 (+2127764us) ts_tablet_manager.cc:1145] Deleting tablet eed058931194480ca1a27130bfa8ee32
0110 02:51:33.332582 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.349185 (+ 16603us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.372325 (+ 23140us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.378348 (+ 6023us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4772,"tablet-delete.queue_time_us":2127504}
I20250110 02:51:33.378839 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 275) took 2174 ms. Trace:
I20250110 02:51:33.378901 2357 rpcz_store.cc:276] 0110 02:51:31.204116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.205124 (+ 1008us) service_pool.cc:225] Handling call
0110 02:51:33.378715 (+2173591us) ts_tablet_manager.cc:1145] Deleting tablet e4726347afd649afa942d74df8c013fe
0110 02:51:33.378724 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.378824 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173458}
I20250110 02:51:33.379243 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 277) took 2019 ms. Trace:
I20250110 02:51:33.379307 2357 rpcz_store.cc:276] 0110 02:51:31.359987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.360045 (+ 58us) service_pool.cc:225] Handling call
0110 02:51:33.379090 (+2019045us) ts_tablet_manager.cc:1145] Deleting tablet 4f015e99f53f48599ac2d7a29220bc39
0110 02:51:33.379098 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.379229 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018871}
I20250110 02:51:33.379516 2357 tablet_replica.cc:331] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.379691 2357 raft_consensus.cc:2238] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.379822 2357 raft_consensus.cc:2267] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.380113 1181 tablet_bootstrap.cc:492] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.380239 1181 ts_tablet_manager.cc:1397] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.001s
I20250110 02:51:33.380904 1181 raft_consensus.cc:357] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.381119 1181 raft_consensus.cc:383] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.381103 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_26326c3a6b1b404bb42e75c5a619be95 [id=78b013e77cf340089467193ef7c17819]) successfully deleted
I20250110 02:51:33.381176 1181 raft_consensus.cc:738] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:33.381369 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e4726347afd649afa942d74df8c013fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4726347afd649afa942d74df8c013fe already in progress: creating tablet
I20250110 02:51:33.381410 1181 consensus_queue.cc:260] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:33.381561 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4f015e99f53f48599ac2d7a29220bc39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f015e99f53f48599ac2d7a29220bc39 already in progress: creating tablet
I20250110 02:51:33.382674 1057 raft_consensus.cc:2463] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:33.383176 711 leader_election.cc:304] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:33.385459 2199 raft_consensus.cc:2798] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.385661 2199 raft_consensus.cc:695] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
I20250110 02:51:33.385892 2199 consensus_queue.cc:237] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.386641 2357 ts_tablet_manager.cc:1905] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:33.388931 996 consensus_peers.cc:487] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.392997 1184 tablet_bootstrap.cc:492] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.393172 1184 ts_tablet_manager.cc:1397] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20250110 02:51:33.393751 1184 raft_consensus.cc:357] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.393901 1184 raft_consensus.cc:383] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.393955 1184 raft_consensus.cc:738] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.394097 1184 consensus_queue.cc:260] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.394645 1184 ts_tablet_manager.cc:1428] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:33.394878 1184 tablet_bootstrap.cc:492] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.396196 1184 tablet_bootstrap.cc:654] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.398972 2357 ts_tablet_manager.cc:1918] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.401393 2357 log.cc:1198] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e804fe33c8ae489bbc4d3765c64b59bd
I20250110 02:51:33.374975 1188 tablet_bootstrap.cc:492] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.401983 1188 ts_tablet_manager.cc:1397] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.010s
I20250110 02:51:33.401886 2357 ts_tablet_manager.cc:1939] T e804fe33c8ae489bbc4d3765c64b59bd P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.402580 1181 ts_tablet_manager.cc:1428] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250110 02:51:33.402709 1188 raft_consensus.cc:357] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.402894 1188 raft_consensus.cc:383] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.402961 1181 tablet_bootstrap.cc:492] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.402954 1188 raft_consensus.cc:738] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.403131 1188 consensus_queue.cc:260] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.403229 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 280) took 1829 ms. Trace:
I20250110 02:51:33.403688 1188 ts_tablet_manager.cc:1428] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 02:51:33.403954 1188 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.403290 2357 rpcz_store.cc:276] 0110 02:51:31.573337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.573390 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:33.379481 (+1806091us) ts_tablet_manager.cc:1145] Deleting tablet e804fe33c8ae489bbc4d3765c64b59bd
0110 02:51:33.379489 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.389349 (+ 9860us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.398860 (+ 9511us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.403214 (+ 4354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793655}
I20250110 02:51:33.404534 1181 tablet_bootstrap.cc:654] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.404520 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e804fe33c8ae489bbc4d3765c64b59bd (table default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]) successfully deleted
I20250110 02:51:33.405382 1188 tablet_bootstrap.cc:654] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.406692 2357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50178 (request call id 281) took 1592 ms. Trace:
I20250110 02:51:33.406767 2357 rpcz_store.cc:276] 0110 02:51:31.814065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.814449 (+ 384us) service_pool.cc:225] Handling call
0110 02:51:33.406584 (+1592135us) ts_tablet_manager.cc:1145] Deleting tablet 6598311f59c6476ba339b8e57961cf67
0110 02:51:33.406591 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.406678 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591999}
W20250110 02:51:33.407176 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6598311f59c6476ba339b8e57961cf67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6598311f59c6476ba339b8e57961cf67 already in progress: creating tablet
I20250110 02:51:33.384776 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:33.387379 2908 raft_consensus.cc:491] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.407392 2908 raft_consensus.cc:513] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.407883 709 leader_election.cc:343] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.407949 2908 leader_election.cc:290] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.408157 1184 tablet_bootstrap.cc:492] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.408290 1184 ts_tablet_manager.cc:1397] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20250110 02:51:33.408613 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.374104 2352 rpcz_store.cc:276] 0110 02:51:28.357618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.358075 (+ 457us) service_pool.cc:225] Handling call
0110 02:51:33.373962 (+5015887us) ts_tablet_manager.cc:1145] Deleting tablet 9e72ed2d31a8467f8c092b48d593c1cc
0110 02:51:33.373971 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.374026 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5015723}
W20250110 02:51:33.409082 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9e72ed2d31a8467f8c092b48d593c1cc because the tablet was not found. No further retry: Not found: Tablet not found: 9e72ed2d31a8467f8c092b48d593c1cc
I20250110 02:51:33.409047 1184 raft_consensus.cc:357] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.409229 1184 raft_consensus.cc:383] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.409281 1184 raft_consensus.cc:738] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.409528 1184 consensus_queue.cc:260] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.410177 1184 ts_tablet_manager.cc:1428] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 02:51:33.410485 1184 tablet_bootstrap.cc:492] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.411163 1188 tablet_bootstrap.cc:492] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.411310 1188 ts_tablet_manager.cc:1397] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250110 02:51:33.410781 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d27f9cb38434646822b6ba1fac53282" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.411834 768 raft_consensus.cc:2388] T 1d27f9cb38434646822b6ba1fac53282 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
I20250110 02:51:33.412565 853 leader_election.cc:304] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.413193 1184 tablet_bootstrap.cc:654] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.413997 2357 tablet_replica.cc:331] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.414211 2357 raft_consensus.cc:2238] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.414402 2357 raft_consensus.cc:2267] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.414886 2357 ts_tablet_manager.cc:1905] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.420157 2378 raft_consensus.cc:3054] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.420627 1188 raft_consensus.cc:357] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.420799 1188 raft_consensus.cc:383] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.420847 1188 raft_consensus.cc:738] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.421064 1188 consensus_queue.cc:260] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.421677 1188 ts_tablet_manager.cc:1428] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250110 02:51:33.421939 1188 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.422423 2378 raft_consensus.cc:2743] T 1d27f9cb38434646822b6ba1fac53282 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.422706 1181 tablet_bootstrap.cc:492] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.423555 1188 tablet_bootstrap.cc:654] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.423753 1181 ts_tablet_manager.cc:1397] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250110 02:51:33.426404 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 324) took 5068 ms. Trace:
I20250110 02:51:33.426456 2357 ts_tablet_manager.cc:1918] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:33.426483 2352 rpcz_store.cc:276] 0110 02:51:28.357814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.358280 (+ 466us) service_pool.cc:225] Handling call
0110 02:51:33.426296 (+5068016us) ts_tablet_manager.cc:1145] Deleting tablet 585320c0970c4097b195caaae33789b5
0110 02:51:33.426306 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.426388 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5067900}
W20250110 02:51:33.426916 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 585320c0970c4097b195caaae33789b5 because the tablet was not found. No further retry: Not found: Tablet not found: 585320c0970c4097b195caaae33789b5
I20250110 02:51:33.427744 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 323) took 5070 ms. Trace:
I20250110 02:51:33.427815 2352 rpcz_store.cc:276] 0110 02:51:28.357694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.358472 (+ 778us) service_pool.cc:225] Handling call
0110 02:51:33.427650 (+5069178us) ts_tablet_manager.cc:1145] Deleting tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11
0110 02:51:33.427658 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.427732 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5068993}
W20250110 02:51:33.428243 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 747ae8cc8c2c4ffa9071e23f7d7c5e11 because the tablet was not found. No further retry: Not found: Tablet not found: 747ae8cc8c2c4ffa9071e23f7d7c5e11
I20250110 02:51:33.430043 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 318) took 5072 ms. Trace:
I20250110 02:51:33.430111 2352 rpcz_store.cc:276] 0110 02:51:28.357272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.358473 (+ 1201us) service_pool.cc:225] Handling call
0110 02:51:33.429978 (+5071505us) ts_tablet_manager.cc:1145] Deleting tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4
0110 02:51:33.429986 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.430030 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5071199}
W20250110 02:51:33.430509 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f5f7c6ca2cd747e78c24fe0b0bf99bf4 because the tablet was not found. No further retry: Not found: Tablet not found: f5f7c6ca2cd747e78c24fe0b0bf99bf4
I20250110 02:51:33.430903 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 325) took 4948 ms. Trace:
I20250110 02:51:33.430960 2352 rpcz_store.cc:276] 0110 02:51:28.482650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.482698 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:33.430824 (+4948126us) ts_tablet_manager.cc:1145] Deleting tablet 1d634dd0c2044f91b304fe3737330d88
0110 02:51:33.430831 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.430892 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4876366}
W20250110 02:51:33.431370 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1d634dd0c2044f91b304fe3737330d88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d634dd0c2044f91b304fe3737330d88 already in progress: creating tablet
I20250110 02:51:33.431612 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 326) took 4939 ms. Trace:
I20250110 02:51:33.431667 2352 rpcz_store.cc:276] 0110 02:51:28.492399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.492448 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:33.431518 (+4939070us) ts_tablet_manager.cc:1145] Deleting tablet 2f0f754966c04074a568c05063591990
0110 02:51:33.431525 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.431600 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4869298}
W20250110 02:51:33.431954 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 2f0f754966c04074a568c05063591990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f0f754966c04074a568c05063591990 already in progress: creating tablet
I20250110 02:51:33.433581 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 327) took 4839 ms. Trace:
I20250110 02:51:33.433640 2352 rpcz_store.cc:276] 0110 02:51:28.594228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.594285 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:33.433500 (+4839215us) ts_tablet_manager.cc:1145] Deleting tablet 1d064a7e8ea34f18b42407903daf7197
0110 02:51:33.433507 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.433568 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4839000}
W20250110 02:51:33.433986 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 1d064a7e8ea34f18b42407903daf7197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d064a7e8ea34f18b42407903daf7197 already in progress: creating tablet
I20250110 02:51:33.436507 2224 ts_tablet_manager.cc:1918] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.436658 2224 log.cc:1198] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f69d0c201c524b7e95c9d3851b8dd8c2
I20250110 02:51:33.437019 2224 ts_tablet_manager.cc:1939] T f69d0c201c524b7e95c9d3851b8dd8c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.437340 1181 raft_consensus.cc:357] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.437635 1181 raft_consensus.cc:383] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.437893 1181 raft_consensus.cc:738] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.438110 2378 raft_consensus.cc:491] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.438206 1181 consensus_queue.cc:260] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.438238 2378 raft_consensus.cc:513] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.438953 2378 leader_election.cc:290] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.441059 1181 ts_tablet_manager.cc:1428] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250110 02:51:33.441442 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.445765 1181 tablet_bootstrap.cc:492] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.447624 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 347) took 5300 ms. Trace:
I20250110 02:51:33.447728 2224 rpcz_store.cc:276] 0110 02:51:28.147368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.147427 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:33.351776 (+5204349us) ts_tablet_manager.cc:1145] Deleting tablet f69d0c201c524b7e95c9d3851b8dd8c2
0110 02:51:33.351785 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.362773 (+ 10988us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.436423 (+ 73650us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.447601 (+ 11178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5204224}
I20250110 02:51:33.448204 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f69d0c201c524b7e95c9d3851b8dd8c2 (table default.Table_8823b1f4fcd7469c8d4eb6264e068b2f [id=50cb2b0aa6604d07a89873488206e3f6]) successfully deleted
I20250110 02:51:33.448712 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 348) took 5301 ms. Trace:
I20250110 02:51:33.449168 2224 rpcz_store.cc:276] 0110 02:51:28.147516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.147581 (+ 65us) service_pool.cc:225] Handling call
0110 02:51:33.448538 (+5300957us) ts_tablet_manager.cc:1145] Deleting tablet d6acae633eb14e5db87b1e36a069422c
0110 02:51:33.448551 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.448692 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5300849}
I20250110 02:51:33.450322 3074 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: acquiring CA information for follower catalog manager: success
W20250110 02:51:33.450901 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d6acae633eb14e5db87b1e36a069422c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6acae633eb14e5db87b1e36a069422c already in progress: creating tablet
I20250110 02:51:33.451890 1188 tablet_bootstrap.cc:492] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.451830 1181 tablet_bootstrap.cc:654] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.452052 1188 ts_tablet_manager.cc:1397] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250110 02:51:33.452687 1188 raft_consensus.cc:357] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.452940 1188 raft_consensus.cc:383] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.453416 1188 raft_consensus.cc:738] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.453693 1188 consensus_queue.cc:260] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.454291 1188 ts_tablet_manager.cc:1428] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.458273 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 349) took 5309 ms. Trace:
I20250110 02:51:33.458412 2224 rpcz_store.cc:276] 0110 02:51:28.148806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.149023 (+ 217us) service_pool.cc:225] Handling call
0110 02:51:33.451421 (+5302398us) ts_tablet_manager.cc:1145] Deleting tablet e848586bec574a529caaf5e498db79cd
0110 02:51:33.458169 (+ 6748us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.458258 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5302237}
W20250110 02:51:33.458952 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e848586bec574a529caaf5e498db79cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e848586bec574a529caaf5e498db79cd already in progress: creating tablet
I20250110 02:51:33.459291 2224 tablet_replica.cc:331] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.459538 2224 raft_consensus.cc:2238] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.459743 2224 raft_consensus.cc:2267] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.460346 2224 ts_tablet_manager.cc:1905] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.439179 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab302fedec4f4c03989b5d3932dc8d4c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.456456 3150 tablet.cc:1721] T 467f87955c4a4d27a6f0483aa109b922 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:33.464718 853 leader_election.cc:304] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.464953 2378 raft_consensus.cc:2743] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.470892 2357 log.cc:1198] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/fe86be26d1b544c08b231f20085d3159
I20250110 02:51:33.471482 2357 ts_tablet_manager.cc:1939] T fe86be26d1b544c08b231f20085d3159 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.473421 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
W20250110 02:51:33.473644 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e55ee565b7f14936a2cb4cbed716c153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e55ee565b7f14936a2cb4cbed716c153 already in progress: creating tablet
W20250110 02:51:33.473821 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because the tablet was not found. No further retry: Not found: Tablet not found: 08705d260c0a462b94ba6b0dc5121123
W20250110 02:51:33.473985 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1edb4e06e40c43438882e1eb622d788e because the tablet was not found. No further retry: Not found: Tablet not found: 1edb4e06e40c43438882e1eb622d788e
W20250110 02:51:33.474154 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
W20250110 02:51:33.474316 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c5601ca24b8f428e8a51cbeec3664901 because the tablet was not found. No further retry: Not found: Tablet not found: c5601ca24b8f428e8a51cbeec3664901
I20250110 02:51:33.474529 911 raft_consensus.cc:3054] T ab302fedec4f4c03989b5d3932dc8d4c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.475056 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because the tablet was not found. No further retry: Not found: Tablet not found: d761ef0d1e5542b1979c29bb2d579489
W20250110 02:51:33.475287 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4de67338a26f4febb5dd7611a72b4723 because the tablet was not found. No further retry: Not found: Tablet not found: 4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:33.475292 917 raft_consensus.cc:3054] T 9395aa9b39d04ac9b6cecf106bc91499 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.477981 3195 tablet.cc:1721] T 467f87955c4a4d27a6f0483aa109b922 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:33.481209 1188 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.483048 1188 tablet_bootstrap.cc:654] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.483053 3149 tablet.cc:1721] T 467f87955c4a4d27a6f0483aa109b922 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 02:51:33.475988 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:33.487002 2289 consensus_queue.cc:1035] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.472791 2357 tablet_replica.cc:331] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.489380 2224 ts_tablet_manager.cc:1918] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.495108 2352 tablet_replica.cc:331] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.495272 2352 raft_consensus.cc:2238] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.495534 2352 raft_consensus.cc:2267] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.495983 2352 ts_tablet_manager.cc:1905] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.496215 1181 tablet_bootstrap.cc:492] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.498481 1181 ts_tablet_manager.cc:1397] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.001s
W20250110 02:51:33.500160 994 consensus_peers.cc:487] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.500607 1181 raft_consensus.cc:357] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.500844 1181 raft_consensus.cc:383] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.500941 1181 raft_consensus.cc:738] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.501154 1181 consensus_queue.cc:260] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.501757 1181 ts_tablet_manager.cc:1428] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:33.502103 2357 raft_consensus.cc:2238] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.502125 1181 tablet_bootstrap.cc:492] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.502312 2357 raft_consensus.cc:2267] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.502802 2357 ts_tablet_manager.cc:1905] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.503599 2224 log.cc:1198] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:33.503983 2224 ts_tablet_manager.cc:1939] T 4de67338a26f4febb5dd7611a72b4723 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.507061 1184 tablet_bootstrap.cc:492] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.507205 1184 ts_tablet_manager.cc:1397] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.000s
I20250110 02:51:33.507264 1188 tablet_bootstrap.cc:492] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.507936 1188 ts_tablet_manager.cc:1397] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250110 02:51:33.507807 1184 raft_consensus.cc:357] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.508005 1181 tablet_bootstrap.cc:654] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.508003 1184 raft_consensus.cc:383] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.508045 1184 raft_consensus.cc:738] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.508383 1184 consensus_queue.cc:260] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.508956 1184 ts_tablet_manager.cc:1428] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:33.509389 853 consensus_peers.cc:487] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0558a2c03f6b42a6befdc48e3af23a6d. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.509289 1188 raft_consensus.cc:357] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.509495 1188 raft_consensus.cc:383] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.510056 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 350) took 5285 ms. Trace:
I20250110 02:51:33.510197 2224 rpcz_store.cc:276] 0110 02:51:28.224158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.224217 (+ 59us) service_pool.cc:225] Handling call
0110 02:51:33.459248 (+5235031us) ts_tablet_manager.cc:1145] Deleting tablet 4de67338a26f4febb5dd7611a72b4723
0110 02:51:33.459258 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.464779 (+ 5521us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.489301 (+ 24522us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.510037 (+ 20736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5234820}
I20250110 02:51:33.510790 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 4de67338a26f4febb5dd7611a72b4723 (table default.Table_2d8cf27cea8e414ebe825a6b96c144c4 [id=efdb6682705641ba8e988cb6034cd5c8]) successfully deleted
I20250110 02:51:33.511181 2224 tablet_replica.cc:331] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.511438 2224 raft_consensus.cc:2238] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.511430 1188 raft_consensus.cc:738] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.511644 2224 raft_consensus.cc:2267] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.512030 1188 consensus_queue.cc:260] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.513351 1188 ts_tablet_manager.cc:1428] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250110 02:51:33.514246 2224 ts_tablet_manager.cc:1905] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.524214 1181 tablet_bootstrap.cc:492] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.524366 1181 ts_tablet_manager.cc:1397] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250110 02:51:33.525161 1181 raft_consensus.cc:357] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.525374 1181 raft_consensus.cc:383] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.525462 1181 raft_consensus.cc:738] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.525692 1181 consensus_queue.cc:260] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.526563 1188 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.529626 2357 ts_tablet_manager.cc:1918] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.529727 2357 log.cc:1198] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:33.530033 2357 ts_tablet_manager.cc:1939] T 5ae4877bba834dbb96ec9675e2aaedd0 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.530609 1184 tablet_bootstrap.cc:492] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.530633 1188 tablet_bootstrap.cc:654] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.531596 1181 ts_tablet_manager.cc:1428] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250110 02:51:33.531853 1181 tablet_bootstrap.cc:492] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.532266 1184 tablet_bootstrap.cc:654] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.533402 1181 tablet_bootstrap.cc:654] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.537796 2224 ts_tablet_manager.cc:1918] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.537950 2224 log.cc:1198] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/7c2c7a313ba94e5ea85e45276658ba04
I20250110 02:51:33.538309 2224 ts_tablet_manager.cc:1939] T 7c2c7a313ba94e5ea85e45276658ba04 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.542264 2352 ts_tablet_manager.cc:1918] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:33.542733 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 351) took 5264 ms. Trace:
I20250110 02:51:33.542861 2224 rpcz_store.cc:276] 0110 02:51:28.278453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.278495 (+ 42us) service_pool.cc:225] Handling call
0110 02:51:33.511141 (+5232646us) ts_tablet_manager.cc:1145] Deleting tablet 7c2c7a313ba94e5ea85e45276658ba04
0110 02:51:33.511150 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.520459 (+ 9309us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.537713 (+ 17254us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.542715 (+ 5002us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3566,"tablet-delete.queue_time_us":5232474}
I20250110 02:51:33.543509 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 7c2c7a313ba94e5ea85e45276658ba04 (table default.Table_c83271627bc9404393dd64f25709c4c6 [id=b076cd9a088f44cc9c0064ea970afd20]) successfully deleted
I20250110 02:51:33.545168 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 352) took 5181 ms. Trace:
I20250110 02:51:33.545297 2224 rpcz_store.cc:276] 0110 02:51:28.363771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.364156 (+ 385us) service_pool.cc:225] Handling call
0110 02:51:33.545058 (+5180902us) ts_tablet_manager.cc:1145] Deleting tablet 9b937188bf2949af8f54bade4e0ad984
0110 02:51:33.545068 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.545152 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5180702}
W20250110 02:51:33.545818 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9b937188bf2949af8f54bade4e0ad984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b937188bf2949af8f54bade4e0ad984 already in progress: creating tablet
I20250110 02:51:33.546068 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 5ae4877bba834dbb96ec9675e2aaedd0 (table default.Table_e04ee12d5f244793a7a44387a6243c1d [id=4ec77430bd4e44aca3a6034e4c8bc6cf]) successfully deleted
I20250110 02:51:33.547066 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 355) took 5182 ms. Trace:
I20250110 02:51:33.547713 2224 rpcz_store.cc:276] 0110 02:51:28.364932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.365164 (+ 232us) service_pool.cc:225] Handling call
0110 02:51:33.546968 (+5181804us) ts_tablet_manager.cc:1145] Deleting tablet 1fb5412b78104d8085878c58b191ab27
0110 02:51:33.546979 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.547047 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181674}
W20250110 02:51:33.548487 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1fb5412b78104d8085878c58b191ab27 because the tablet was not found. No further retry: Not found: Tablet not found: 1fb5412b78104d8085878c58b191ab27
I20250110 02:51:33.549073 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 357) took 5184 ms. Trace:
I20250110 02:51:33.554872 1184 tablet_bootstrap.cc:492] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.555040 1184 ts_tablet_manager.cc:1397] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250110 02:51:33.555657 1184 raft_consensus.cc:357] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.555859 1184 raft_consensus.cc:383] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.555931 1184 raft_consensus.cc:738] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.556186 1184 consensus_queue.cc:260] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.556712 1184 ts_tablet_manager.cc:1428] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:33.557173 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dc6b2654f86e4686b8a3034de02d11bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc6b2654f86e4686b8a3034de02d11bf already in progress: creating tablet
I20250110 02:51:33.557508 1184 tablet_bootstrap.cc:492] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.542351 2352 log.cc:1198] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f0d4d6e02b1d4334a9993d320217f1f8
I20250110 02:51:33.561869 2352 ts_tablet_manager.cc:1939] T f0d4d6e02b1d4334a9993d320217f1f8 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.562762 1188 tablet_bootstrap.cc:492] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.554584 2905 raft_consensus.cc:491] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.562952 1188 ts_tablet_manager.cc:1397] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250110 02:51:33.563027 2905 raft_consensus.cc:513] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.563587 1188 raft_consensus.cc:357] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.563846 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.563865 1188 raft_consensus.cc:383] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.564181 1188 raft_consensus.cc:738] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
W20250110 02:51:33.564294 994 leader_election.cc:343] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.564432 1188 consensus_queue.cc:260] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.564522 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.564703 772 raft_consensus.cc:2463] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.564997 1188 ts_tablet_manager.cc:1428] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.565048 996 leader_election.cc:304] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.565255 2905 leader_election.cc:290] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.565543 1188 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.565919 2905 raft_consensus.cc:2798] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.566040 2905 raft_consensus.cc:491] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.566150 2905 raft_consensus.cc:3054] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.568692 2905 raft_consensus.cc:513] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.570055 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.570363 2905 leader_election.cc:290] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.570751 2224 rpcz_store.cc:276] 0110 02:51:28.365044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.365317 (+ 273us) service_pool.cc:225] Handling call
0110 02:51:33.548965 (+5183648us) ts_tablet_manager.cc:1145] Deleting tablet cdb726d2c9a04963b92a3fb625cf84f3
0110 02:51:33.548978 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.549052 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5183552}
I20250110 02:51:33.571571 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7064f3f16f6144a4b624eff2ad3ccf62" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:33.571805 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because the tablet was not found. No further retry: Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
W20250110 02:51:33.571985 994 leader_election.cc:343] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.570231 772 raft_consensus.cc:3054] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.576455 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet c28f191afa2a43608bbd74322bec97a6 because the tablet was not found. No further retry: Not found: Tablet not found: c28f191afa2a43608bbd74322bec97a6
W20250110 02:51:33.577793 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e848586bec574a529caaf5e498db79cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e848586bec574a529caaf5e498db79cd already in progress: creating tablet
I20250110 02:51:33.580737 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 354) took 5215 ms. Trace:
I20250110 02:51:33.581208 1188 tablet_bootstrap.cc:654] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.581498 2224 rpcz_store.cc:276] 0110 02:51:28.364871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.365324 (+ 453us) service_pool.cc:225] Handling call
0110 02:51:33.572188 (+5206864us) ts_tablet_manager.cc:1145] Deleting tablet 04e05a7554a14805bdcbd1824a5bd9d5
0110 02:51:33.572361 (+ 173us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.580720 (+ 8359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3456,"tablet-delete.queue_time_us":5206688}
I20250110 02:51:33.583736 1184 tablet_bootstrap.cc:654] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.581444 1181 tablet_bootstrap.cc:492] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.584234 1181 ts_tablet_manager.cc:1397] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
W20250110 02:51:33.584558 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because the tablet was not found. No further retry: Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:33.584470 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 356) took 5219 ms. Trace:
I20250110 02:51:33.585271 2224 rpcz_store.cc:276] 0110 02:51:28.364988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.365523 (+ 535us) service_pool.cc:225] Handling call
0110 02:51:33.584386 (+5218863us) ts_tablet_manager.cc:1145] Deleting tablet 65ae1c19d2dc4778a1cebddbd505f907
0110 02:51:33.584396 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.584454 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5218724}
I20250110 02:51:33.586900 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:33.589179 2357 tablet_replica.cc:331] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.592252 2357 raft_consensus.cc:2238] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.592437 2357 raft_consensus.cc:2267] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.592533 2224 tablet_replica.cc:331] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.592550 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:33.592831 2224 raft_consensus.cc:2238] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.593022 2357 ts_tablet_manager.cc:1905] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.592782 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:33.593967 2224 raft_consensus.cc:2267] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:33.590545 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d6acae633eb14e5db87b1e36a069422c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6acae633eb14e5db87b1e36a069422c already in progress: creating tablet
I20250110 02:51:33.593250 1181 raft_consensus.cc:357] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.594198 1181 raft_consensus.cc:383] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.594255 1181 raft_consensus.cc:738] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:33.594251 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1d064a7e8ea34f18b42407903daf7197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d064a7e8ea34f18b42407903daf7197 already in progress: creating tablet
W20250110 02:51:33.594458 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2f0f754966c04074a568c05063591990 because the tablet was not found. No further retry: Not found: Tablet not found: 2f0f754966c04074a568c05063591990
I20250110 02:51:33.594524 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
W20250110 02:51:33.594614 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1d634dd0c2044f91b304fe3737330d88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d634dd0c2044f91b304fe3737330d88 already in progress: creating tablet
W20250110 02:51:33.594811 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 82bd4307e5fe4f61b9308005cf3446e1 because the tablet was not found. No further retry: Not found: Tablet not found: 82bd4307e5fe4f61b9308005cf3446e1
I20250110 02:51:33.594957 2224 ts_tablet_manager.cc:1905] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:33.594978 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fc1324eeac9446279304f442e2fad488 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc1324eeac9446279304f442e2fad488 already in progress: creating tablet
W20250110 02:51:33.595150 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f9b379b357194d729108428cbaebd51a because the tablet was not found. No further retry: Not found: Tablet not found: f9b379b357194d729108428cbaebd51a
W20250110 02:51:33.595297 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
W20250110 02:51:33.595597 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 6555b0a4491046bf8f5cd4ee406c3955 because the tablet was not found. No further retry: Not found: Tablet not found: 6555b0a4491046bf8f5cd4ee406c3955
W20250110 02:51:33.595789 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7c2c7a313ba94e5ea85e45276658ba04 because the tablet was not found. No further retry: Not found: Tablet not found: 7c2c7a313ba94e5ea85e45276658ba04
I20250110 02:51:33.595539 1181 consensus_queue.cc:260] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.596009 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9b937188bf2949af8f54bade4e0ad984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b937188bf2949af8f54bade4e0ad984 already in progress: creating tablet
W20250110 02:51:33.596220 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f0d4d6e02b1d4334a9993d320217f1f8 because the tablet was not found. No further retry: Not found: Tablet not found: f0d4d6e02b1d4334a9993d320217f1f8
W20250110 02:51:33.596455 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
W20250110 02:51:33.596704 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 2e25f9643ac0476a90c93fb9aeb921a3 because the tablet was not found. No further retry: Not found: Tablet not found: 2e25f9643ac0476a90c93fb9aeb921a3
W20250110 02:51:33.597314 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 33f9b3e74ee2431a85a22fefaacfa40e because the tablet was not found. No further retry: Not found: Tablet not found: 33f9b3e74ee2431a85a22fefaacfa40e
W20250110 02:51:33.597584 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 24034fb3da6e47978cdc9b02e714e30d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24034fb3da6e47978cdc9b02e714e30d already in progress: creating tablet
W20250110 02:51:33.597851 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 55edfa05075b4edcb70ef29e5ee413c6 because the tablet was not found. No further retry: Not found: Tablet not found: 55edfa05075b4edcb70ef29e5ee413c6
W20250110 02:51:33.598084 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b45b91d084a44c1e8cdf1553967878e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b45b91d084a44c1e8cdf1553967878e2 already in progress: creating tablet
W20250110 02:51:33.598340 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dba0cb7970f1453dac256b255ed7cac0 because the tablet was not found. No further retry: Not found: Tablet not found: dba0cb7970f1453dac256b255ed7cac0
W20250110 02:51:33.598654 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a7d4f03a8c66446ea7c6d178cc7ef80a because the tablet was not found. No further retry: Not found: Tablet not found: a7d4f03a8c66446ea7c6d178cc7ef80a
W20250110 02:51:33.598999 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because the tablet was not found. No further retry: Not found: Tablet not found: 676f5f457d9e4028bfd0b707604c42f5
W20250110 02:51:33.599298 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because the tablet was not found. No further retry: Not found: Tablet not found: d00698a0a9ee4e5da69a745f4a3aa8b4
W20250110 02:51:33.599552 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d4f1d916a5ea48708c2c1a61c92fe137 because the tablet was not found. No further retry: Not found: Tablet not found: d4f1d916a5ea48708c2c1a61c92fe137
W20250110 02:51:33.599800 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b43bb790e85343308a1bb1e1110bfb9c because the tablet was not found. No further retry: Not found: Tablet not found: b43bb790e85343308a1bb1e1110bfb9c
W20250110 02:51:33.600071 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 8e79175853474cbea951f5a5686bf630 because the tablet was not found. No further retry: Not found: Tablet not found: 8e79175853474cbea951f5a5686bf630
W20250110 02:51:33.600296 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 213939cf13354ce380fb7cda5b38676f because the tablet was not found. No further retry: Not found: Tablet not found: 213939cf13354ce380fb7cda5b38676f
W20250110 02:51:33.600849 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bc8444c1381a45d6b2fee30da971bd47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8444c1381a45d6b2fee30da971bd47 already in progress: creating tablet
I20250110 02:51:33.601114 1181 ts_tablet_manager.cc:1428] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20250110 02:51:33.601102 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3fb805b3e537466ab7fe0185421ad5aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fb805b3e537466ab7fe0185421ad5aa already in progress: creating tablet
W20250110 02:51:33.601325 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 818dcbf83303412d97b097913bcd36c3 because the tablet was not found. No further retry: Not found: Tablet not found: 818dcbf83303412d97b097913bcd36c3
W20250110 02:51:33.601559 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet d424b4da66184bd19eccefad80621516 because the tablet was not found. No further retry: Not found: Tablet not found: d424b4da66184bd19eccefad80621516
W20250110 02:51:33.601836 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 65ae1c19d2dc4778a1cebddbd505f907 because the tablet was not found. No further retry: Not found: Tablet not found: 65ae1c19d2dc4778a1cebddbd505f907
I20250110 02:51:33.604851 1181 tablet_bootstrap.cc:492] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.607288 2357 ts_tablet_manager.cc:1918] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.595295 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:33.608255 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:33.609850 1443 catalog_manager.cc:5526] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:33.616194 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 328) took 5021 ms. Trace:
I20250110 02:51:33.616304 2352 rpcz_store.cc:276] 0110 02:51:28.594342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.594530 (+ 188us) service_pool.cc:225] Handling call
0110 02:51:33.434363 (+4839833us) ts_tablet_manager.cc:1145] Deleting tablet f0d4d6e02b1d4334a9993d320217f1f8
0110 02:51:33.495071 (+ 60708us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.539573 (+ 44502us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.542194 (+ 2621us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.616143 (+ 73949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4839686}
I20250110 02:51:33.616755 1181 tablet_bootstrap.cc:654] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.616735 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09]) successfully deleted
I20250110 02:51:33.618350 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 329) took 5023 ms. Trace:
I20250110 02:51:33.618463 2352 rpcz_store.cc:276] 0110 02:51:28.594478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.594701 (+ 223us) service_pool.cc:225] Handling call
0110 02:51:33.616559 (+5021858us) ts_tablet_manager.cc:1145] Deleting tablet 071bbce1706c43b6bb690c7de1ead3fe
0110 02:51:33.618205 (+ 1646us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.618314 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5021748}
I20250110 02:51:33.618814 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 330) took 4927 ms. Trace:
I20250110 02:51:33.618909 2352 rpcz_store.cc:276] 0110 02:51:28.691772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.691832 (+ 60us) service_pool.cc:225] Handling call
0110 02:51:33.618684 (+4926852us) ts_tablet_manager.cc:1145] Deleting tablet f5e88d1bbc304b4ebbc849a7d783052e
0110 02:51:33.618695 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.618747 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4926655}
W20250110 02:51:33.619133 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 071bbce1706c43b6bb690c7de1ead3fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 071bbce1706c43b6bb690c7de1ead3fe already in progress: creating tablet
I20250110 02:51:33.619215 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 331) took 4922 ms. Trace:
I20250110 02:51:33.619294 2352 rpcz_store.cc:276] 0110 02:51:28.696621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.696668 (+ 47us) service_pool.cc:225] Handling call
0110 02:51:33.619115 (+4922447us) ts_tablet_manager.cc:1145] Deleting tablet f14f06ba7d0f44418693e36f6bc6d888
0110 02:51:33.619126 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.619173 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4922300}
W20250110 02:51:33.619517 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f5e88d1bbc304b4ebbc849a7d783052e because the tablet was not found. No further retry: Not found: Tablet not found: f5e88d1bbc304b4ebbc849a7d783052e
I20250110 02:51:33.619619 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 332) took 4922 ms. Trace:
I20250110 02:51:33.619714 2352 rpcz_store.cc:276] 0110 02:51:28.697403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.697478 (+ 75us) service_pool.cc:225] Handling call
0110 02:51:33.619526 (+4922048us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:33.619536 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.619585 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4921916}
I20250110 02:51:33.620007 2352 tablet_replica.cc:331] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.620211 2352 raft_consensus.cc:2238] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.620373 2352 raft_consensus.cc:2267] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:33.621479 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f14f06ba7d0f44418693e36f6bc6d888 because the tablet was not found. No further retry: Not found: Tablet not found: f14f06ba7d0f44418693e36f6bc6d888
W20250110 02:51:33.621709 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 because the tablet was not found. No further retry: Not found: Tablet not found: b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:33.622299 2357 log.cc:1198] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/a0ddab9e847445999ec97eac4352d9c2
I20250110 02:51:33.622623 2357 ts_tablet_manager.cc:1939] T a0ddab9e847445999ec97eac4352d9c2 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.622705 1443 catalog_manager.cc:5526] T ac5dab241cd74f329a57e6f19e95a7b6 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:33.623791 2352 ts_tablet_manager.cc:1905] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.623546 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet a0ddab9e847445999ec97eac4352d9c2 (table default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]) successfully deleted
I20250110 02:51:33.624383 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:33.628402 2224 ts_tablet_manager.cc:1918] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.628556 2224 log.cc:1198] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/a0ddab9e847445999ec97eac4352d9c2
I20250110 02:51:33.628942 2224 ts_tablet_manager.cc:1939] T a0ddab9e847445999ec97eac4352d9c2 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.629218 3074 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 8bc934cc708045e9858ee8f8e64660cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 02:51:33.633409 2352 ts_tablet_manager.cc:1918] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.633510 2352 log.cc:1198] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/bb595013e809480a814cb9c7aed856e6
I20250110 02:51:33.633800 2352 ts_tablet_manager.cc:1939] T bb595013e809480a814cb9c7aed856e6 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.634483 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 341) took 4528 ms. Trace:
I20250110 02:51:33.634606 2352 rpcz_store.cc:276] 0110 02:51:29.105853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.112233 (+ 6380us) service_pool.cc:225] Handling call
0110 02:51:33.619946 (+4507713us) ts_tablet_manager.cc:1145] Deleting tablet bb595013e809480a814cb9c7aed856e6
0110 02:51:33.619954 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.630844 (+ 10890us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.633324 (+ 2480us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.634445 (+ 1121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4507461}
I20250110 02:51:33.636452 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 358) took 5271 ms. Trace:
I20250110 02:51:33.636528 2224 rpcz_store.cc:276] 0110 02:51:28.365127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.365685 (+ 558us) service_pool.cc:225] Handling call
0110 02:51:33.592299 (+5226614us) ts_tablet_manager.cc:1145] Deleting tablet a0ddab9e847445999ec97eac4352d9c2
0110 02:51:33.592309 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.613451 (+ 21142us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.628301 (+ 14850us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.636434 (+ 8133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5226473}
I20250110 02:51:33.637068 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 359) took 5152 ms. Trace:
I20250110 02:51:33.637185 2224 rpcz_store.cc:276] 0110 02:51:28.484311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.484364 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:33.636882 (+5152518us) ts_tablet_manager.cc:1145] Deleting tablet 1d634dd0c2044f91b304fe3737330d88
0110 02:51:33.636891 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.637053 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152313}
I20250110 02:51:33.637840 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 360) took 5145 ms. Trace:
I20250110 02:51:33.637971 2224 rpcz_store.cc:276] 0110 02:51:28.492495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.492538 (+ 43us) service_pool.cc:225] Handling call
0110 02:51:33.637740 (+5145202us) ts_tablet_manager.cc:1145] Deleting tablet 2f0f754966c04074a568c05063591990
0110 02:51:33.637748 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.637824 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5145013}
I20250110 02:51:33.638455 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 361) took 5053 ms. Trace:
I20250110 02:51:33.638566 2224 rpcz_store.cc:276] 0110 02:51:28.585066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.585134 (+ 68us) service_pool.cc:225] Handling call
0110 02:51:33.638352 (+5053218us) ts_tablet_manager.cc:1145] Deleting tablet 1d064a7e8ea34f18b42407903daf7197
0110 02:51:33.638360 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.638442 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5053026}
I20250110 02:51:33.638944 2352 tablet_replica.cc:331] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.639092 2224 tablet_replica.cc:331] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.639240 2352 raft_consensus.cc:2238] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.639559 2352 raft_consensus.cc:2267] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.639722 1184 tablet_bootstrap.cc:492] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.639904 1184 ts_tablet_manager.cc:1397] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.005s
I20250110 02:51:33.640110 2352 ts_tablet_manager.cc:1905] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.639310 2224 raft_consensus.cc:2238] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.640529 2224 raft_consensus.cc:2267] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.640594 1184 raft_consensus.cc:357] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.640762 1184 raft_consensus.cc:383] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.640821 1184 raft_consensus.cc:738] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.640992 1184 consensus_queue.cc:260] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.641553 1184 ts_tablet_manager.cc:1428] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:33.641815 1184 tablet_bootstrap.cc:492] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.635015 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]) successfully deleted
I20250110 02:51:33.639788 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet a0ddab9e847445999ec97eac4352d9c2 (table default.Table_3e713a5d93834ea3be890e774e266d83 [id=873710046a0e46cbb034750e6432efc9]) successfully deleted
W20250110 02:51:33.642443 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1d634dd0c2044f91b304fe3737330d88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d634dd0c2044f91b304fe3737330d88 already in progress: creating tablet
I20250110 02:51:33.641602 2224 ts_tablet_manager.cc:1905] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:33.642695 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 2f0f754966c04074a568c05063591990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f0f754966c04074a568c05063591990 already in progress: creating tablet
W20250110 02:51:33.642917 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1d064a7e8ea34f18b42407903daf7197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d064a7e8ea34f18b42407903daf7197 already in progress: creating tablet
I20250110 02:51:33.643103 1184 tablet_bootstrap.cc:654] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:33.643160 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f69d0c201c524b7e95c9d3851b8dd8c2 because the tablet was not found. No further retry: Not found: Tablet not found: f69d0c201c524b7e95c9d3851b8dd8c2
W20250110 02:51:33.643344 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7564208da99449cd97338a41b263e750 because the tablet was not found. No further retry: Not found: Tablet not found: 7564208da99449cd97338a41b263e750
W20250110 02:51:33.643548 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7e554d00d05b4fa7a14c9b6159f9d726 because the tablet was not found. No further retry: Not found: Tablet not found: 7e554d00d05b4fa7a14c9b6159f9d726
W20250110 02:51:33.643752 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9507a5283d8f4d07a293ba474c642298 because the tablet was not found. No further retry: Not found: Tablet not found: 9507a5283d8f4d07a293ba474c642298
W20250110 02:51:33.643926 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f63d69d04b494a848653636c9e259105 because the tablet was not found. No further retry: Not found: Tablet not found: f63d69d04b494a848653636c9e259105
W20250110 02:51:33.644088 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3b6d57f3e9a44d178300d3505d504854 because the tablet was not found. No further retry: Not found: Tablet not found: 3b6d57f3e9a44d178300d3505d504854
W20250110 02:51:33.645020 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because the tablet was not found. No further retry: Not found: Tablet not found: eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:33.645697 2352 ts_tablet_manager.cc:1918] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.645814 2352 log.cc:1198] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f6636a3b7f2c4e2fb0ef76f9ab25dcfb
I20250110 02:51:33.646157 2352 ts_tablet_manager.cc:1939] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.646878 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 342) took 4515 ms. Trace:
I20250110 02:51:33.646975 2352 rpcz_store.cc:276] 0110 02:51:29.130921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.141669 (+ 10748us) service_pool.cc:225] Handling call
0110 02:51:33.638883 (+4497214us) ts_tablet_manager.cc:1145] Deleting tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb
0110 02:51:33.638893 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.643321 (+ 4428us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.645612 (+ 2291us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.646837 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4496939}
W20250110 02:51:33.647114 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1390f867a3a8436092d1920daff4c5b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1390f867a3a8436092d1920daff4c5b0
I20250110 02:51:33.647045 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 3aee71c51a96482fbb527576867bbaa9 (table default.Table_a99fe640359046aa8ff288db7a687676 [id=c9a367ed70c742a18d7dc76e72b37300]). Expected version 1 got 0
I20250110 02:51:33.647553 2352 tablet_replica.cc:331] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.647362 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]) successfully deleted
I20250110 02:51:33.647761 2352 raft_consensus.cc:2238] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:33.647823 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4aa78a9ef5eb443da0c8983a7190440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78a9ef5eb443da0c8983a7190440b already in progress: creating tablet
I20250110 02:51:33.647938 2352 raft_consensus.cc:2267] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:33.648118 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 27a7885a80414f2193400672aa16a911 because the tablet was not found. No further retry: Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
I20250110 02:51:33.648221 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:33.648445 2352 ts_tablet_manager.cc:1905] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.657802 2352 ts_tablet_manager.cc:1918] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.657893 2352 log.cc:1198] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/3ca895eca14748c19fdb7d363499b544
I20250110 02:51:33.658165 2352 ts_tablet_manager.cc:1939] T 3ca895eca14748c19fdb7d363499b544 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.658836 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 348) took 3051 ms. Trace:
I20250110 02:51:33.658916 2352 rpcz_store.cc:276] 0110 02:51:30.607051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:30.607096 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:33.647497 (+3040401us) ts_tablet_manager.cc:1145] Deleting tablet 3ca895eca14748c19fdb7d363499b544
0110 02:51:33.647506 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.654987 (+ 7481us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.657732 (+ 2745us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.658819 (+ 1087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3040190}
I20250110 02:51:33.659564 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 364) took 2482 ms. Trace:
I20250110 02:51:33.659626 2352 rpcz_store.cc:276] 0110 02:51:31.177045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.177101 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:33.659478 (+2482377us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:33.659486 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.659550 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456076}
I20250110 02:51:33.659981 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 365) took 2482 ms. Trace:
I20250110 02:51:33.660037 2352 rpcz_store.cc:276] 0110 02:51:31.177565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178011 (+ 446us) service_pool.cc:225] Handling call
0110 02:51:33.659917 (+2481906us) ts_tablet_manager.cc:1145] Deleting tablet 6555b0a4491046bf8f5cd4ee406c3955
0110 02:51:33.659925 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.659971 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456438}
I20250110 02:51:33.660354 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 366) took 2482 ms. Trace:
I20250110 02:51:33.660411 2352 rpcz_store.cc:276] 0110 02:51:31.177669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178160 (+ 491us) service_pool.cc:225] Handling call
0110 02:51:33.660300 (+2482140us) ts_tablet_manager.cc:1145] Deleting tablet cf711522866d46439d7ee69aa63d7f39
0110 02:51:33.660307 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.660345 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2454929}
I20250110 02:51:33.661015 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 367) took 2483 ms. Trace:
I20250110 02:51:33.661083 2352 rpcz_store.cc:276] 0110 02:51:31.177805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178271 (+ 466us) service_pool.cc:225] Handling call
0110 02:51:33.660945 (+2482674us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:33.660953 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.660999 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2455502}
W20250110 02:51:33.661337 994 consensus_peers.cc:487] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.661402 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 368) took 2483 ms. Trace:
I20250110 02:51:33.661461 2352 rpcz_store.cc:276] 0110 02:51:31.177959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178352 (+ 393us) service_pool.cc:225] Handling call
0110 02:51:33.661347 (+2482995us) ts_tablet_manager.cc:1145] Deleting tablet 4d970e2fe72b4727968ca605b3e772cc
0110 02:51:33.661354 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.661392 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2455859}
I20250110 02:51:33.661801 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 369) took 2483 ms. Trace:
I20250110 02:51:33.661870 2352 rpcz_store.cc:276] 0110 02:51:31.178613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178669 (+ 56us) service_pool.cc:225] Handling call
0110 02:51:33.661727 (+2483058us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:33.661736 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.661791 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456188}
I20250110 02:51:33.662175 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 370) took 2483 ms. Trace:
I20250110 02:51:33.662247 2352 rpcz_store.cc:276] 0110 02:51:31.178722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178786 (+ 64us) service_pool.cc:225] Handling call
0110 02:51:33.662122 (+2483336us) ts_tablet_manager.cc:1145] Deleting tablet 19e4af23720048fe98b3231ba402d45c
0110 02:51:33.662128 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.662166 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456536}
I20250110 02:51:33.663097 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 371) took 2484 ms. Trace:
I20250110 02:51:33.663167 2352 rpcz_store.cc:276] 0110 02:51:31.178862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.178930 (+ 68us) service_pool.cc:225] Handling call
0110 02:51:33.662519 (+2483589us) ts_tablet_manager.cc:1145] Deleting tablet 9289de0630804f9aa60fdaf5f3f15f68
0110 02:51:33.662526 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.663082 (+ 556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456883}
I20250110 02:51:33.663334 2906 raft_consensus.cc:491] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.663439 2906 raft_consensus.cc:513] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.663877 2289 consensus_queue.cc:1035] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.663964 2906 leader_election.cc:290] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:33.664299 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.664474 2352 tablet_replica.cc:331] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.664700 2352 raft_consensus.cc:2238] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
W20250110 02:51:33.664682 994 leader_election.cc:343] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.664918 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.664994 2352 raft_consensus.cc:2267] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.665073 771 raft_consensus.cc:2463] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.665452 2352 ts_tablet_manager.cc:1905] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.665962 996 leader_election.cc:304] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.666448 2289 raft_consensus.cc:2798] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.666523 2289 raft_consensus.cc:491] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.666574 2289 raft_consensus.cc:3054] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.668979 2289 raft_consensus.cc:513] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.669519 2289 leader_election.cc:290] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:33.670528 2352 ts_tablet_manager.cc:1918] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.670617 2352 log.cc:1198] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/0558a2c03f6b42a6befdc48e3af23a6d
W20250110 02:51:33.670835 996 consensus_peers.cc:487] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb595013e809480a814cb9c7aed856e6. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.670782 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:33.670921 2352 ts_tablet_manager.cc:1939] T 0558a2c03f6b42a6befdc48e3af23a6d P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:33.671226 994 leader_election.cc:343] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.671257 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248055ca83de4e62a6b7e908c53d3d9a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.671422 771 raft_consensus.cc:3054] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.671604 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 372) took 2476 ms. Trace:
I20250110 02:51:33.671667 2352 rpcz_store.cc:276] 0110 02:51:31.194820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.194869 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:33.664437 (+2469568us) ts_tablet_manager.cc:1145] Deleting tablet 0558a2c03f6b42a6befdc48e3af23a6d
0110 02:51:33.664445 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.667998 (+ 3553us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.670462 (+ 2464us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.671592 (+ 1130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2454396}
I20250110 02:51:33.672034 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 374) took 2469 ms. Trace:
I20250110 02:51:33.672089 2352 rpcz_store.cc:276] 0110 02:51:31.202532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.202606 (+ 74us) service_pool.cc:225] Handling call
0110 02:51:33.671955 (+2469349us) ts_tablet_manager.cc:1145] Deleting tablet e4726347afd649afa942d74df8c013fe
0110 02:51:33.671962 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.672023 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457023}
I20250110 02:51:33.672420 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 373) took 2470 ms. Trace:
I20250110 02:51:33.672472 2352 rpcz_store.cc:276] 0110 02:51:31.201900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.201955 (+ 55us) service_pool.cc:225] Handling call
0110 02:51:33.672346 (+2470391us) ts_tablet_manager.cc:1145] Deleting tablet eed058931194480ca1a27130bfa8ee32
0110 02:51:33.672353 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.672409 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2455686}
I20250110 02:51:33.672782 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 375) took 2313 ms. Trace:
I20250110 02:51:33.672835 2352 rpcz_store.cc:276] 0110 02:51:31.358877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.358953 (+ 76us) service_pool.cc:225] Handling call
0110 02:51:33.672710 (+2313757us) ts_tablet_manager.cc:1145] Deleting tablet 4f015e99f53f48599ac2d7a29220bc39
0110 02:51:33.672717 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.672771 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313575}
I20250110 02:51:33.673099 2352 tablet_replica.cc:331] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.673249 2352 raft_consensus.cc:2238] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.673360 2352 raft_consensus.cc:2267] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.673797 2352 ts_tablet_manager.cc:1905] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.674927 771 raft_consensus.cc:2463] T 248055ca83de4e62a6b7e908c53d3d9a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.675408 996 leader_election.cc:304] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.675838 772 raft_consensus.cc:2463] T 7064f3f16f6144a4b624eff2ad3ccf62 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.675928 2289 raft_consensus.cc:2798] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.676323 2289 raft_consensus.cc:695] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.676795 2289 consensus_queue.cc:237] T 248055ca83de4e62a6b7e908c53d3d9a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.676820 1184 tablet_bootstrap.cc:492] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.677798 1184 ts_tablet_manager.cc:1397] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250110 02:51:33.678339 996 leader_election.cc:304] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.678396 1184 raft_consensus.cc:357] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.678565 1184 raft_consensus.cc:383] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.678643 2905 raft_consensus.cc:2798] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.678730 1184 raft_consensus.cc:738] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.678779 2905 raft_consensus.cc:695] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.678990 1184 consensus_queue.cc:260] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.678992 2905 consensus_queue.cc:237] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.679380 2352 ts_tablet_manager.cc:1918] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.679462 2352 log.cc:1198] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e804fe33c8ae489bbc4d3765c64b59bd
I20250110 02:51:33.679513 1184 ts_tablet_manager.cc:1428] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.679771 2352 ts_tablet_manager.cc:1939] T e804fe33c8ae489bbc4d3765c64b59bd P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.679806 1184 tablet_bootstrap.cc:492] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.680408 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 378) took 2109 ms. Trace:
I20250110 02:51:33.680483 2352 rpcz_store.cc:276] 0110 02:51:31.570902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.570943 (+ 41us) service_pool.cc:225] Handling call
0110 02:51:33.673068 (+2102125us) ts_tablet_manager.cc:1145] Deleting tablet e804fe33c8ae489bbc4d3765c64b59bd
0110 02:51:33.673074 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.676230 (+ 3156us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.679309 (+ 3079us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.680387 (+ 1078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2078984}
I20250110 02:51:33.680860 2352 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43876 (request call id 379) took 1870 ms. Trace:
I20250110 02:51:33.680928 2352 rpcz_store.cc:276] 0110 02:51:31.810886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.810947 (+ 61us) service_pool.cc:225] Handling call
0110 02:51:33.680777 (+1869830us) ts_tablet_manager.cc:1145] Deleting tablet 6598311f59c6476ba339b8e57961cf67
0110 02:51:33.680785 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.680848 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1869667}
I20250110 02:51:33.681244 2352 tablet_replica.cc:331] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:33.681392 2352 raft_consensus.cc:2238] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.681646 1184 tablet_bootstrap.cc:654] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.681646 2352 raft_consensus.cc:2267] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.682072 2352 ts_tablet_manager.cc:1905] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.682852 2456 consensus_queue.cc:1035] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.684289 2378 raft_consensus.cc:491] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.684391 2378 raft_consensus.cc:513] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.684861 2378 leader_election.cc:290] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:33.685953 853 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: dba0cb7970f1453dac256b255ed7cac0
W20250110 02:51:33.686462 853 leader_election.cc:343] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: dba0cb7970f1453dac256b255ed7cac0
I20250110 02:51:33.686550 853 leader_election.cc:304] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.686926 2378 raft_consensus.cc:2743] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.690930 2352 ts_tablet_manager.cc:1918] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:33.691594 2352 log.cc:1198] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/fe86be26d1b544c08b231f20085d3159
I20250110 02:51:33.691969 1184 tablet_bootstrap.cc:492] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.692081 1184 ts_tablet_manager.cc:1397] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250110 02:51:33.692358 2352 ts_tablet_manager.cc:1939] T fe86be26d1b544c08b231f20085d3159 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:33.692690 1184 raft_consensus.cc:357] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.692890 1184 raft_consensus.cc:383] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.692962 1184 raft_consensus.cc:738] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.693153 1184 consensus_queue.cc:260] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.694123 1184 ts_tablet_manager.cc:1428] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.694384 1184 tablet_bootstrap.cc:492] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.695914 1184 tablet_bootstrap.cc:654] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:33.649086 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bcc9a4a842914a6f878d01194cd9a5f7 because the tablet was not found. No further retry: Not found: Tablet not found: bcc9a4a842914a6f878d01194cd9a5f7
I20250110 02:51:33.659323 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 3ca895eca14748c19fdb7d363499b544 (table default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]) successfully deleted
W20250110 02:51:33.698629 1433 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:45139, user_credentials={real_user=slave}} blocked reactor thread for 49555.2us
W20250110 02:51:33.698877 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because the tablet was not found. No further retry: Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
W20250110 02:51:33.698940 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f6df074974974d81a83420e7a3a78070 because the tablet was not found. No further retry: Not found: Tablet not found: f6df074974974d81a83420e7a3a78070
W20250110 02:51:33.699072 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6555b0a4491046bf8f5cd4ee406c3955 because the tablet was not found. No further retry: Not found: Tablet not found: 6555b0a4491046bf8f5cd4ee406c3955
W20250110 02:51:33.699153 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 1b7ec8c7891b4a2c9fcb3cc2eeedd29a because the tablet was not found. No further retry: Not found: Tablet not found: 1b7ec8c7891b4a2c9fcb3cc2eeedd29a
W20250110 02:51:33.699260 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cf711522866d46439d7ee69aa63d7f39 because the tablet was not found. No further retry: Not found: Tablet not found: cf711522866d46439d7ee69aa63d7f39
W20250110 02:51:33.699352 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because the tablet was not found. No further retry: Not found: Tablet not found: aa6057de9da04f2c82b8f25c5b4e2fe4
W20250110 02:51:33.699434 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because the tablet was not found. No further retry: Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
W20250110 02:51:33.699556 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because the tablet was not found. No further retry: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5
W20250110 02:51:33.699599 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because the tablet was not found. No further retry: Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc
W20250110 02:51:33.699746 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because the tablet was not found. No further retry: Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
W20250110 02:51:33.699764 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 071bbce1706c43b6bb690c7de1ead3fe because the tablet was not found. No further retry: Not found: Tablet not found: 071bbce1706c43b6bb690c7de1ead3fe
W20250110 02:51:33.699905 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because the tablet was not found. No further retry: Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c
W20250110 02:51:33.699965 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bb595013e809480a814cb9c7aed856e6 because the tablet was not found. No further retry: Not found: Tablet not found: bb595013e809480a814cb9c7aed856e6
W20250110 02:51:33.700078 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because the tablet was not found. No further retry: Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:33.700209 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 0558a2c03f6b42a6befdc48e3af23a6d (table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]) successfully deleted
W20250110 02:51:33.700352 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e4726347afd649afa942d74df8c013fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4726347afd649afa942d74df8c013fe already in progress: creating tablet
W20250110 02:51:33.700510 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet eed058931194480ca1a27130bfa8ee32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eed058931194480ca1a27130bfa8ee32 already in progress: creating tablet
W20250110 02:51:33.700649 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4f015e99f53f48599ac2d7a29220bc39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f015e99f53f48599ac2d7a29220bc39 already in progress: creating tablet
I20250110 02:51:33.700796 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e804fe33c8ae489bbc4d3765c64b59bd (table default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]) successfully deleted
W20250110 02:51:33.700989 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6598311f59c6476ba339b8e57961cf67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6598311f59c6476ba339b8e57961cf67 already in progress: creating tablet
I20250110 02:51:33.701159 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
W20250110 02:51:33.701367 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e55ee565b7f14936a2cb4cbed716c153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e55ee565b7f14936a2cb4cbed716c153 already in progress: creating tablet
I20250110 02:51:33.701519 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
W20250110 02:51:33.701653 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because the tablet was not found. No further retry: Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
W20250110 02:51:33.701822 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because the tablet was not found. No further retry: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb
I20250110 02:51:33.701884 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 9e28d8769f364e10aad54b60fb221f35 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
W20250110 02:51:33.702008 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because the tablet was not found. No further retry: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5
W20250110 02:51:33.702179 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because the tablet was not found. No further retry: Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
W20250110 02:51:33.705225 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because the tablet was not found. No further retry: Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
W20250110 02:51:33.705464 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because the tablet was not found. No further retry: Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
W20250110 02:51:33.705718 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because the tablet was not found. No further retry: Not found: Tablet not found: d00698a0a9ee4e5da69a745f4a3aa8b4
W20250110 02:51:33.705924 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because the tablet was not found. No further retry: Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
W20250110 02:51:33.706204 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d424b4da66184bd19eccefad80621516 because the tablet was not found. No further retry: Not found: Tablet not found: d424b4da66184bd19eccefad80621516
W20250110 02:51:33.706557 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7564208da99449cd97338a41b263e750 because the tablet was not found. No further retry: Not found: Tablet not found: 7564208da99449cd97338a41b263e750
W20250110 02:51:33.706799 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because the tablet was not found. No further retry: Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:33.706995 2378 raft_consensus.cc:491] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.707111 2378 raft_consensus.cc:513] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.707655 2378 leader_election.cc:290] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:33.707772 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet e015a41a03f84c91a45e46e4b7b5e7a9 because the tablet was not found. No further retry: Not found: Tablet not found: e015a41a03f84c91a45e46e4b7b5e7a9
W20250110 02:51:33.707947 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d761ef0d1e5542b1979c29bb2d579489 because the tablet was not found. No further retry: Not found: Tablet not found: d761ef0d1e5542b1979c29bb2d579489
W20250110 02:51:33.708094 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because the tablet was not found. No further retry: Not found: Tablet not found: 6e0b2ec1d5814144bc58261e6c6e8d35
W20250110 02:51:33.708248 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because the tablet was not found. No further retry: Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:33.708425 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae4877bba834dbb96ec9675e2aaedd0" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.702312 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet c1ade70d7fe34776bef3c5853f8dde74 (table default.Table_83c9397ae7e84b17a964ab1d664307b2 [id=88412101305b4d0a8dea71c08ff391aa]). Expected version 1 got 0
W20250110 02:51:33.709519 853 leader_election.cc:343] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:33.709573 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet fd02e9f8cd0748a0a6a3202fbe390ad1 (table default.Table_6f66870093e04c74a3762a4c14c6bc91 [id=72f48ddababf496785d9361a0ff5a10c]). Expected version 1 got 0
I20250110 02:51:33.709618 853 leader_election.cc:304] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.709731 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 656aa84a130c42768e969b9322fbd8a5 (table default.Table_7f95dbc0488e40178420f7e412eac048 [id=497a56d44a114093934dbb9fde1b9f8a]). Expected version 1 got 0
I20250110 02:51:33.710330 2378 raft_consensus.cc:2743] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:33.710616 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because the tablet was not found. No further retry: Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
W20250110 02:51:33.710868 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8f15078fc22f4b32a48c7190b36e2192 because the tablet was not found. No further retry: Not found: Tablet not found: 8f15078fc22f4b32a48c7190b36e2192
I20250110 02:51:33.711165 1188 tablet_bootstrap.cc:492] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.711283 1188 ts_tablet_manager.cc:1397] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.000s
I20250110 02:51:33.711798 1188 raft_consensus.cc:357] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.711961 1188 raft_consensus.cc:383] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.712008 1188 raft_consensus.cc:738] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.712184 1188 consensus_queue.cc:260] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.712661 1188 ts_tablet_manager.cc:1428] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 02:51:33.719487 2224 ts_tablet_manager.cc:1918] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:33.719627 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4f1d7940a1574af1804d5f5139ca7494 (table default.Table_b4223b31a5974abb9725bceb8026a8ae [id=4e2fb95ceb8c43fc920b1cd9e73557df]). Expected version 1 got 0
I20250110 02:51:33.720229 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet dc1901385a534e5dad20954b763329a0 (table default.Table_3e90520ccabf4d1d9f02e80796522910 [id=72c128c9bf8849638e64f4eb99ec4c7e]). Expected version 1 got 0
I20250110 02:51:33.725586 1188 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.726943 1188 tablet_bootstrap.cc:654] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.728348 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 2312c1ee4a134caaa68980a224dd283e (table default.Table_4e121eab5d694b9f970bacc67c1a3841 [id=1a070f9c16754f38b88eae8f08d33c2d]). Expected version 1 got 0
I20250110 02:51:33.728905 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0d63ccced9a24eb19dcb8f9b3e7269ee (table default.Table_7376d50f0b824f59a76abea37a60a400 [id=77f1fac11ce54f608198ea6f401d0047]). Expected version 1 got 0
I20250110 02:51:33.729368 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 7064f3f16f6144a4b624eff2ad3ccf62 (table default.Table_5594db9942c346058f56f1abc8b3262d [id=bc34df1b32364b59b69a031c43574246]). Expected version 1 got 0
I20250110 02:51:33.729549 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:33.729880 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b9fb775de0154409a0d92990a51d1ee0 (table default.Table_74bf258302164b9eaccfd22680070c50 [id=ed3a9f13f7794c3f9ee6917bf6ecff25]). Expected version 1 got 0
I20250110 02:51:33.730182 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet b5f162de87044005bdafe93292ff71b8 (table default.Table_d0ba98c5ff8f4aff8a2f4a636e81b4ed [id=f733132df99b481ea5a2117fe65a7fc4]). Expected version 1 got 0
I20250110 02:51:33.730484 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet deaa3527aa074e6ba94960c672aee028 (table default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2 [id=6333eee82fd747f8a89b10e31b04e5e9]). Expected version 1 got 0
I20250110 02:51:33.730801 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:33.731563 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 39cfc508c1b84299a2f16d307bb25354 (table default.Table_c4a5fd4dc8a94e108deada6468c9fa8e [id=a14528ef7bd24ede9db29c29df81c11c]). Expected version 1 got 0
I20250110 02:51:33.732143 1443 catalog_manager.cc:5526] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:33.732560 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 4bd2283e7ed7456dae466a8582587eb7 (table default.Table_5a006cecde524c1dadb83743a5782b86 [id=71a78c3f9fca4c6da82ace0c6d3df656]). Expected version 2 got 0
I20250110 02:51:33.732798 1443 catalog_manager.cc:5526] T 54b538100977456794c9519bbdffa7e7 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:33.733131 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 54b538100977456794c9519bbdffa7e7 (table default.Table_f3aa7e42b29e455aa70415be887fded5 [id=fa0cfecca12c4ddcade02cc7a553edf1]). Expected version 2 got 0
I20250110 02:51:33.733690 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_614d56e4a7544c27aa6f21776b2138f3 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 3 got 0
I20250110 02:51:33.733961 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
W20250110 02:51:33.723054 994 consensus_peers.cc:487] T 4bd2283e7ed7456dae466a8582587eb7 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.724123 2289 consensus_queue.cc:1035] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 02:51:33.731269 3152 consensus_queue.cc:1035] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.738917 2224 log.cc:1198] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f0d4d6e02b1d4334a9993d320217f1f8
I20250110 02:51:33.742050 2224 ts_tablet_manager.cc:1939] T f0d4d6e02b1d4334a9993d320217f1f8 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.742919 1181 tablet_bootstrap.cc:492] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.743908 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 362) took 5153 ms. Trace:
I20250110 02:51:33.747800 1184 tablet_bootstrap.cc:492] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.747916 1184 ts_tablet_manager.cc:1397] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.005s
I20250110 02:51:33.744268 1181 ts_tablet_manager.cc:1397] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.003s
I20250110 02:51:33.748185 2224 rpcz_store.cc:276] 0110 02:51:28.590031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.590076 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:33.639050 (+5048974us) ts_tablet_manager.cc:1145] Deleting tablet f0d4d6e02b1d4334a9993d320217f1f8
0110 02:51:33.639059 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.657910 (+ 18851us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.719377 (+ 61467us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.743892 (+ 24515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5048732}
I20250110 02:51:33.748975 2224 tablet_replica.cc:331] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.748903 1181 raft_consensus.cc:357] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.749354 1181 raft_consensus.cc:383] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.749429 1181 raft_consensus.cc:738] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.749661 2224 raft_consensus.cc:2238] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.750244 1184 raft_consensus.cc:357] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.750439 1184 raft_consensus.cc:383] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.750491 1184 raft_consensus.cc:738] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.749621 1181 consensus_queue.cc:260] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.750742 2224 raft_consensus.cc:2267] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.750942 1181 ts_tablet_manager.cc:1428] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:33.751147 1181 tablet_bootstrap.cc:492] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.751672 2224 ts_tablet_manager.cc:1905] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.752503 1188 tablet_bootstrap.cc:492] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.752636 1188 ts_tablet_manager.cc:1397] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20250110 02:51:33.752643 1181 tablet_bootstrap.cc:654] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.753509 1184 consensus_queue.cc:260] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.754841 1184 ts_tablet_manager.cc:1428] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250110 02:51:33.755057 1184 tablet_bootstrap.cc:492] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.756523 1184 tablet_bootstrap.cc:654] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.759069 2905 raft_consensus.cc:491] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.759289 2905 raft_consensus.cc:513] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.760177 2905 leader_election.cc:290] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.760426 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.760605 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.760741 772 raft_consensus.cc:2463] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.753257 1188 raft_consensus.cc:357] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.760823 994 leader_election.cc:343] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.760881 1188 raft_consensus.cc:383] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.760948 1188 raft_consensus.cc:738] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.761070 996 leader_election.cc:304] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.761279 2905 raft_consensus.cc:2798] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.761358 2905 raft_consensus.cc:491] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.761443 2905 raft_consensus.cc:3054] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.765084 2905 raft_consensus.cc:513] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.765841 2905 leader_election.cc:290] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.766112 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:33.766321 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e54ae5d7e864aff974a107f6762f446" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:33.761158 1188 consensus_queue.cc:260] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.766482 772 raft_consensus.cc:3054] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.766628 994 leader_election.cc:343] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.766829 1188 ts_tablet_manager.cc:1428] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250110 02:51:33.767052 1188 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.768281 1188 tablet_bootstrap.cc:654] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.768714 772 raft_consensus.cc:2463] T 4e54ae5d7e864aff974a107f6762f446 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.769064 996 leader_election.cc:304] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.769498 2289 raft_consensus.cc:2798] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.769655 2289 raft_consensus.cc:695] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.769846 2289 consensus_queue.cc:237] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:33.777015 996 consensus_peers.cc:487] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed058931194480ca1a27130bfa8ee32. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.778494 1188 tablet_bootstrap.cc:492] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:33.778630 1188 ts_tablet_manager.cc:1397] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20250110 02:51:33.779402 1188 raft_consensus.cc:357] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.779563 1188 raft_consensus.cc:383] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.779614 1188 raft_consensus.cc:738] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.779867 1188 consensus_queue.cc:260] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.780414 1188 ts_tablet_manager.cc:1428] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:33.780664 1188 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:33.781848 1188 tablet_bootstrap.cc:654] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.785981 2224 ts_tablet_manager.cc:1918] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:33.786085 2224 log.cc:1198] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/071bbce1706c43b6bb690c7de1ead3fe
I20250110 02:51:33.748993 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f0d4d6e02b1d4334a9993d320217f1f8 (table default.Table_5bef6387b90c461e8266e7b3052ee939 [id=996c474b298141e48b7a828a367d0e09]) successfully deleted
I20250110 02:51:33.791270 1184 tablet_bootstrap.cc:492] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.791391 1184 ts_tablet_manager.cc:1397] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.006s
I20250110 02:51:33.791993 1184 raft_consensus.cc:357] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.792174 1184 raft_consensus.cc:383] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.734357 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 0059fbcbfb7a4f239a5947f702d21c0f (table default.Table_bbc18e2b41314211bf9690b0b60739a1 [id=cba096e296394b3ca68bc08f9770efcd]). Expected version 1 got 0
I20250110 02:51:33.793771 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 467f87955c4a4d27a6f0483aa109b922 (table default.Table_deca6d7b78fe4d78a2d5fd5a9e1d38bd [id=b58434c397244de09dd22be49baeeb09]). Expected version 3 got 0
I20250110 02:51:33.794361 1443 catalog_manager.cc:5526] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8899dd1aeb384e4ea4523f5fd225c888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20250110 02:51:33.797432 2289 consensus_queue.cc:1035] T 1d27f9cb38434646822b6ba1fac53282 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.799183 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c]). Expected version 1 got 0
I20250110 02:51:33.799546 1443 catalog_manager.cc:5618] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771) does not have the latest schema for tablet 09c029deb4c54ce68d560059020f0bba (table default.Table_fb3102e06de047a68de0adb4febe7735 [id=db8c2210559d44c6bd41a41f4d49ee21]). Expected version 1 got 0
I20250110 02:51:33.800268 1449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet be5ac15f4e774b7e854e4c5b4737b591
I20250110 02:51:33.800657 1449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 179) took 2406 ms. Trace:
I20250110 02:51:33.800740 1449 rpcz_store.cc:276] 0110 02:51:31.394621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.161605 (+1766984us) service_pool.cc:225] Handling call
0110 02:51:33.162128 (+ 523us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:33.162272 (+ 144us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:33.162341 (+ 69us) catalog_manager.cc:2762] Locking tablets
0110 02:51:33.162489 (+ 148us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:33.282515 (+120026us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:33.282593 (+ 78us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:33.800220 (+517627us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:33.800631 (+ 411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.176562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.176632 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.176636 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:33.176638 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.176726 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.176733 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.176737 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:33.176914 (+ 177us) tablet.cc:698] Row locks acquired
0110 02:51:33.176918 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:33.176943 (+ 25us) write_op.cc:270] Start()
0110 02:51:33.176964 (+ 21us) write_op.cc:276] Timestamp: P: 1736477493176917 usec, L: 0
0110 02:51:33.176969 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.177090 (+ 121us) log.cc:844] Serialized 725 byte log entry
0110 02:51:33.277535 (+100445us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.282016 (+ 4481us) write_op.cc:301] APPLY: starting
0110 02:51:33.282035 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.282082 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.282142 (+ 60us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.282210 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.282250 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:33.282257 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:33.282331 (+ 74us) log.cc:844] Serialized 29 byte log entry
0110 02:51:33.282412 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.282432 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:33.282439 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:33.282516 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.282519 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4402,"apply.run_cpu_time_us":500,"apply.run_wall_time_us":500,"num_ops":2,"prepare.queue_time_us":13498,"prepare.run_cpu_time_us":537,"prepare.run_wall_time_us":609,"replication_time_us":100525,"spinlock_wait_cycles":46336}]]}
I20250110 02:51:33.801980 2224 ts_tablet_manager.cc:1939] T 071bbce1706c43b6bb690c7de1ead3fe P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.802695 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 363) took 5206 ms. Trace:
I20250110 02:51:33.802772 2224 rpcz_store.cc:276] 0110 02:51:28.596427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:28.596478 (+ 51us) service_pool.cc:225] Handling call
0110 02:51:33.748930 (+5152452us) ts_tablet_manager.cc:1145] Deleting tablet 071bbce1706c43b6bb690c7de1ead3fe
0110 02:51:33.748938 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.783263 (+ 34325us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.785914 (+ 2651us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.802679 (+ 16765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5152267}
I20250110 02:51:33.803822 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.804016 740 tablet_service.cc:1514] Processing DeleteTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.804201 2357 tablet_replica.cc:331] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.804363 2357 raft_consensus.cc:2238] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.804484 2357 raft_consensus.cc:2267] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.804478 879 tablet_service.cc:1514] Processing DeleteTablet for tablet be5ac15f4e774b7e854e4c5b4737b591 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.805104 2357 ts_tablet_manager.cc:1905] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:33.805341 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet be5ac15f4e774b7e854e4c5b4737b591 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5ac15f4e774b7e854e4c5b4737b591 already in progress: creating tablet
I20250110 02:51:33.806085 1181 tablet_bootstrap.cc:492] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:33.806177 1181 ts_tablet_manager.cc:1397] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.006s
I20250110 02:51:33.806798 1181 raft_consensus.cc:357] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.806949 1181 raft_consensus.cc:383] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.806989 1181 raft_consensus.cc:738] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.807132 1181 consensus_queue.cc:260] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.807595 1181 ts_tablet_manager.cc:1428] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:33.808121 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 071bbce1706c43b6bb690c7de1ead3fe (table default.Table_add5accf5d07401eadb673123fda5ae6 [id=51b598525c1147198940e02ef5dcdf85]) successfully deleted
I20250110 02:51:33.812352 1181 tablet_bootstrap.cc:492] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:33.813004 2357 ts_tablet_manager.cc:1918] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.813401 2357 log.cc:1198] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/be5ac15f4e774b7e854e4c5b4737b591
W20250110 02:51:33.813452 1433 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 70e53b838be846c7b3ac44dc75aa3730 (table default.Table_08102d27b4484be78d1304da43bcfd31 [id=35affb86ced149b7a2182d622a13c08c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 70e53b838be846c7b3ac44dc75aa3730 consensus state has no leader
I20250110 02:51:33.814092 2357 ts_tablet_manager.cc:1939] T be5ac15f4e774b7e854e4c5b4737b591 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
W20250110 02:51:33.814160 996 consensus_peers.cc:487] T 213939cf13354ce380fb7cda5b38676f P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 213939cf13354ce380fb7cda5b38676f. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.814438 1181 tablet_bootstrap.cc:654] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.815109 2224 tablet_replica.cc:331] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.815335 2224 raft_consensus.cc:2238] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.815708 2224 raft_consensus.cc:2267] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.816344 2224 ts_tablet_manager.cc:1905] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.816977 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet be5ac15f4e774b7e854e4c5b4737b591 (table default.Table_98b0cdca19ab460ca64c35faf958d9bb [id=1553b9ccf48849ae941241a322cd105d]) successfully deleted
I20250110 02:51:33.792220 1184 raft_consensus.cc:738] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.819419 1184 consensus_queue.cc:260] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.822119 1184 ts_tablet_manager.cc:1428] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250110 02:51:33.822350 1184 tablet_bootstrap.cc:492] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.823545 1184 tablet_bootstrap.cc:654] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.825126 3224 raft_consensus.cc:491] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.825240 3224 raft_consensus.cc:513] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.825827 3224 leader_election.cc:290] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.826325 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff57f894a5b54dee91452cbb4cb38b0c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.826679 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff57f894a5b54dee91452cbb4cb38b0c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.826820 711 leader_election.cc:343] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.827091 709 leader_election.cc:343] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.827181 709 leader_election.cc:304] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.827580 3224 raft_consensus.cc:2743] T ff57f894a5b54dee91452cbb4cb38b0c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.839082 1184 tablet_bootstrap.cc:492] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.839277 1184 ts_tablet_manager.cc:1397] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20250110 02:51:33.839900 1184 raft_consensus.cc:357] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.840119 1184 raft_consensus.cc:383] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.840201 1184 raft_consensus.cc:738] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.840718 1184 consensus_queue.cc:260] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.841569 1184 ts_tablet_manager.cc:1428] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250110 02:51:33.841888 994 consensus_peers.cc:487] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.842312 1184 tablet_bootstrap.cc:492] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.843463 1184 tablet_bootstrap.cc:654] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.847632 2224 ts_tablet_manager.cc:1918] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.847744 2224 log.cc:1198] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/bb595013e809480a814cb9c7aed856e6
I20250110 02:51:33.848112 2224 ts_tablet_manager.cc:1939] T bb595013e809480a814cb9c7aed856e6 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.865545 3224 raft_consensus.cc:491] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.865660 3224 raft_consensus.cc:513] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.866142 3224 leader_election.cc:290] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.866663 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc408e62034459815486648565e7ce" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.867139 709 leader_election.cc:343] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.864455 3152 consensus_queue.cc:1035] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250110 02:51:33.867694 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2abc408e62034459815486648565e7ce" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:33.868922 711 leader_election.cc:343] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.869027 711 leader_election.cc:304] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.869240 3224 raft_consensus.cc:2743] T 2abc408e62034459815486648565e7ce P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.869362 1444 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_8f3acd38298944faabb07e4e3b27e9ec [id=7b6faf943b984803b3086d2128a42462] alter complete (version 1)
I20250110 02:51:33.848819 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 371) took 4742 ms. Trace:
I20250110 02:51:33.882611 2224 rpcz_store.cc:276] 0110 02:51:29.106353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.109651 (+ 3298us) service_pool.cc:225] Handling call
0110 02:51:33.805203 (+4695552us) ts_tablet_manager.cc:1145] Deleting tablet bb595013e809480a814cb9c7aed856e6
0110 02:51:33.812147 (+ 6944us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.833738 (+ 21591us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.847547 (+ 13809us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.848800 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4695313}
I20250110 02:51:33.883411 2224 tablet_replica.cc:331] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.883564 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet bb595013e809480a814cb9c7aed856e6 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]) successfully deleted
I20250110 02:51:33.884028 2224 raft_consensus.cc:2238] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.884200 2224 raft_consensus.cc:2267] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.884725 2224 ts_tablet_manager.cc:1905] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.886575 3152 raft_consensus.cc:491] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.886694 3152 raft_consensus.cc:513] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.887163 3152 leader_election.cc:290] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.887692 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55edfa05075b4edcb70ef29e5ee413c6" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.887826 1439 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_42e03c7535b24cf7981f2a93a5d719f3 [id=bae70feea8bb4af3b1e7601399e19534] alter complete (version 1)
W20250110 02:51:33.888561 994 leader_election.cc:343] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.888753 996 leader_election.cc:343] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 55edfa05075b4edcb70ef29e5ee413c6
I20250110 02:51:33.888818 996 leader_election.cc:304] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888; no voters: 85f9597077934f1a9814d284ed06d13d, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.889240 3152 raft_consensus.cc:2743] T 55edfa05075b4edcb70ef29e5ee413c6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.891043 2378 raft_consensus.cc:491] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.891211 2378 raft_consensus.cc:513] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.891779 2378 leader_election.cc:290] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.893721 891 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c031451104d5a93a1672b4b1918c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.893846 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c4042c38de4969889f4821176fc7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.893966 894 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.894121 885 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.894243 884 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.894364 887 tablet_service.cc:1514] Processing DeleteTablet for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 with delete_type TABLET_DATA_DELETED (replaced by tablet ab302fedec4f4c03989b5d3932dc8d4c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.894500 895 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:33.895095 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because the tablet was not found. No further retry: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5
W20250110 02:51:33.895285 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because the tablet was not found. No further retry: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb
W20250110 02:51:33.895421 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f0d4d6e02b1d4334a9993d320217f1f8 because the tablet was not found. No further retry: Not found: Tablet not found: f0d4d6e02b1d4334a9993d320217f1f8
W20250110 02:51:33.895556 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because the tablet was not found. No further retry: Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
W20250110 02:51:33.895892 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because the tablet was not found. No further retry: Not found: Tablet not found: d00698a0a9ee4e5da69a745f4a3aa8b4
W20250110 02:51:33.896029 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because the tablet was not found. No further retry: Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
W20250110 02:51:33.896158 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because the tablet was not found. No further retry: Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
I20250110 02:51:33.897198 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:33.897611 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4de67338a26f4febb5dd7611a72b4723 because the tablet was not found. No further retry: Not found: Tablet not found: 4de67338a26f4febb5dd7611a72b4723
W20250110 02:51:33.898312 853 leader_election.cc:343] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: f63d69d04b494a848653636c9e259105
W20250110 02:51:33.899284 853 leader_election.cc:343] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: f63d69d04b494a848653636c9e259105
I20250110 02:51:33.899410 853 leader_election.cc:304] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.899644 2378 raft_consensus.cc:2743] T f63d69d04b494a848653636c9e259105 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.900779 1184 tablet_bootstrap.cc:492] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.900964 1184 ts_tablet_manager.cc:1397] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.030s
W20250110 02:51:33.901448 1456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 114) took 1956 ms (client timeout 1000 ms). Trace:
I20250110 02:51:33.901525 1184 raft_consensus.cc:357] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.901680 1184 raft_consensus.cc:383] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.901715 1184 raft_consensus.cc:738] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.901888 1184 consensus_queue.cc:260] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.902364 1184 ts_tablet_manager.cc:1428] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 02:51:33.902642 1184 tablet_bootstrap.cc:492] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.902922 894 tablet_service.cc:1514] Processing DeleteTablet for tablet 08705d260c0a462b94ba6b0dc5121123 with delete_type TABLET_DATA_DELETED (replaced by tablet a7d4f03a8c66446ea7c6d178cc7ef80a) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903028 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903105 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903177 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903237 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 452223af5f6a4836a0f582d394a9453e with delete_type TABLET_DATA_DELETED (replaced by tablet fc1324eeac9446279304f442e2fad488) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903307 887 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903504 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 7564208da99449cd97338a41b263e750 with delete_type TABLET_DATA_DELETED (replaced by tablet 6598311f59c6476ba339b8e57961cf67) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903684 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.903872 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d424b4da66184bd19eccefad80621516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:33.904042 1456 rpcz_store.cc:269] 0110 02:51:31.945388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:32.700181 (+754793us) service_pool.cc:225] Handling call
0110 02:51:33.901390 (+1201209us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:32.799553 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:32.799658 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:32.799664 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:32.799667 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:32.799778 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:32.799791 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:32.799796 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:32.799854 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:32.799856 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:32.799887 (+ 31us) write_op.cc:270] Start()
0110 02:51:32.799934 (+ 47us) write_op.cc:276] Timestamp: P: 1736477492799840 usec, L: 0
0110 02:51:32.799938 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:32.800099 (+ 161us) log.cc:844] Serialized 356 byte log entry
0110 02:51:32.804953 (+ 4854us) op_driver.cc:464] REPLICATION: finished
0110 02:51:32.805079 (+ 126us) write_op.cc:301] APPLY: starting
0110 02:51:32.805093 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:32.805124 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:32.805154 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:32.805214 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:32.805242 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:32.805250 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:32.805314 (+ 64us) log.cc:844] Serialized 23 byte log entry
0110 02:51:32.805364 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.805380 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:32.805386 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:32.805448 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.805452 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 02:51:33.101320 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.101457 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.101466 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:33.101471 (+ 5us) tablet.cc:659] Decoding operations
0110 02:51:33.101576 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.101587 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.101592 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:33.101652 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:33.101656 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:33.101691 (+ 35us) write_op.cc:270] Start()
0110 02:51:33.101723 (+ 32us) write_op.cc:276] Timestamp: P: 1736477493101669 usec, L: 0
0110 02:51:33.101726 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.174427 (+ 72701us) log.cc:844] Serialized 356 byte log entry
0110 02:51:33.174589 (+ 162us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 000000000096ceda 00007f953116c632 00007f95324f4eb5 00007f95324f39d9 00007f95324fbe00 00007f95324fbc0d 0000000000a71ecc 00007f9525939b64 00007f952593d18b 00007f952593d008 0000000000a71ecc
0110 02:51:33.178455 (+ 3866us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.178644 (+ 189us) write_op.cc:301] APPLY: starting
0110 02:51:33.178656 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.178698 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.178703 (+ 5us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.201385 (+ 22682us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.201414 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:33.201424 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:33.201566 (+ 142us) log.cc:844] Serialized 23 byte log entry
0110 02:51:33.201623 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.201638 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:33.201645 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:33.201717 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.201723 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8454,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":359,"apply.run_wall_time_us":358,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":628,"raft.queue_time_us":343,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":193,"replication_time_us":4987}],["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":458,"apply.run_wall_time_us":23068,"num_ops":1,"prepare.queue_time_us":139465,"prepare.run_cpu_time_us":1051,"prepare.run_wall_time_us":73642,"raft.queue_time_us":215,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":391,"replication_time_us":76706,"spinlock_wait_cycles":166768640}]]}
W20250110 02:51:33.905561 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 08705d260c0a462b94ba6b0dc5121123 because the tablet was not found. No further retry: Not found: Tablet not found: 08705d260c0a462b94ba6b0dc5121123
W20250110 02:51:33.905788 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because the tablet was not found. No further retry: Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
W20250110 02:51:33.905952 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because the tablet was not found. No further retry: Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:33.905825 1456 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_428c37996d7b406083d3f703508d1729" } new_table_name: "default.Table_8e012f0c2f3146a08f11fe095f25d0fa" modify_external_catalogs: true
W20250110 02:51:33.906798 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3ca895eca14748c19fdb7d363499b544 because the tablet was not found. No further retry: Not found: Tablet not found: 3ca895eca14748c19fdb7d363499b544
W20250110 02:51:33.906960 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 452223af5f6a4836a0f582d394a9453e because the tablet was not found. No further retry: Not found: Tablet not found: 452223af5f6a4836a0f582d394a9453e
W20250110 02:51:33.907104 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fe86be26d1b544c08b231f20085d3159 because the tablet was not found. No further retry: Not found: Tablet not found: fe86be26d1b544c08b231f20085d3159
W20250110 02:51:33.907256 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7564208da99449cd97338a41b263e750 because the tablet was not found. No further retry: Not found: Tablet not found: 7564208da99449cd97338a41b263e750
W20250110 02:51:33.907390 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7c2c7a313ba94e5ea85e45276658ba04 because the tablet was not found. No further retry: Not found: Tablet not found: 7c2c7a313ba94e5ea85e45276658ba04
W20250110 02:51:33.907542 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d424b4da66184bd19eccefad80621516 because the tablet was not found. No further retry: Not found: Tablet not found: d424b4da66184bd19eccefad80621516
I20250110 02:51:33.908134 1184 tablet_bootstrap.cc:654] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:33.910681 996 consensus_peers.cc:487] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ca895eca14748c19fdb7d363499b544. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:33.926139 3152 consensus_queue.cc:1035] T 2b2b5e3c14244418912f344b7519995d P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.928077 3152 consensus_queue.cc:1035] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.932067 2224 ts_tablet_manager.cc:1918] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.935528 1184 tablet_bootstrap.cc:492] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:33.936040 1184 ts_tablet_manager.cc:1397] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.011s
I20250110 02:51:33.936276 1106 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:33.936842 1184 raft_consensus.cc:357] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.937676 1184 raft_consensus.cc:383] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.937736 1184 raft_consensus.cc:738] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.938084 1184 consensus_queue.cc:260] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:33.938539 2378 raft_consensus.cc:491] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.938655 2378 raft_consensus.cc:513] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.939270 2378 leader_election.cc:290] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.939620 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.940026 1184 ts_tablet_manager.cc:1428] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:33.940260 1184 tablet_bootstrap.cc:492] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.940461 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ab2376576448deb87f607b3e5a4f1c" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.940948 853 leader_election.cc:304] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.941196 2378 raft_consensus.cc:2743] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.943114 1184 tablet_bootstrap.cc:654] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.946756 2224 log.cc:1198] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f6636a3b7f2c4e2fb0ef76f9ab25dcfb
I20250110 02:51:33.947340 2224 ts_tablet_manager.cc:1939] T f6636a3b7f2c4e2fb0ef76f9ab25dcfb P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:33.950424 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 372) took 4826 ms. Trace:
I20250110 02:51:33.951320 2224 rpcz_store.cc:276] 0110 02:51:29.123492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:29.123538 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:33.883371 (+4759833us) ts_tablet_manager.cc:1145] Deleting tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb
0110 02:51:33.883380 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.907951 (+ 24571us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.931967 (+ 24016us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:33.950408 (+ 18441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4759678}
I20250110 02:51:33.952061 2224 tablet_replica.cc:331] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:33.952208 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb (table default.Table_dde6fdcfbda44b7e9a8f0ab4a501a565 [id=0c2d2e22fc0d44308dc8185393986584]) successfully deleted
I20250110 02:51:33.955322 2289 raft_consensus.cc:491] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.955423 2289 raft_consensus.cc:513] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.955904 2289 leader_election.cc:290] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.956326 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.956527 768 raft_consensus.cc:2463] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.956789 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.956983 996 leader_election.cc:304] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:33.957278 2905 raft_consensus.cc:2798] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.957363 2905 raft_consensus.cc:491] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.957432 2905 raft_consensus.cc:3054] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.959959 2905 raft_consensus.cc:513] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.960412 2905 leader_election.cc:290] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.960813 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
W20250110 02:51:33.960947 994 leader_election.cc:343] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.960994 768 raft_consensus.cc:3054] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:33.961478 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f3e8cc6ac4442690b9d062bf1c2dc4" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:33.961534 912 raft_consensus.cc:3054] T 5ae4877bba834dbb96ec9675e2aaedd0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.961827 994 leader_election.cc:343] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.963332 768 raft_consensus.cc:2463] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:33.963765 996 leader_election.cc:304] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.964013 2905 raft_consensus.cc:2798] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:33.964118 2905 raft_consensus.cc:695] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:33.964345 2905 consensus_queue.cc:237] T d1f3e8cc6ac4442690b9d062bf1c2dc4 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.965572 3152 consensus_queue.cc:1035] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:33.970568 1450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 36c0aaf2050f481daa8cdac4761fe705
I20250110 02:51:33.971233 1450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 182) took 2558 ms. Trace:
I20250110 02:51:33.971336 1450 rpcz_store.cc:276] 0110 02:51:31.412606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.227335 (+1814729us) service_pool.cc:225] Handling call
0110 02:51:33.283873 (+ 56538us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.284226 (+ 353us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:33.284355 (+ 129us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:33.284418 (+ 63us) catalog_manager.cc:2762] Locking tablets
0110 02:51:33.284559 (+ 141us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:33.326442 (+ 41883us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:33.326536 (+ 94us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:33.970507 (+643971us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:33.971178 (+ 671us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.290694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.290804 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.290810 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:33.290812 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.290916 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.290927 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.290931 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:33.291030 (+ 99us) tablet.cc:698] Row locks acquired
0110 02:51:33.291033 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:33.291064 (+ 31us) write_op.cc:270] Start()
0110 02:51:33.291090 (+ 26us) write_op.cc:276] Timestamp: P: 1736477493291024 usec, L: 0
0110 02:51:33.291093 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.291226 (+ 133us) log.cc:844] Serialized 725 byte log entry
0110 02:51:33.310512 (+ 19286us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.324966 (+ 14454us) write_op.cc:301] APPLY: starting
0110 02:51:33.324984 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.325019 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.325050 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.325104 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.325139 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:33.325145 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:33.325211 (+ 66us) log.cc:844] Serialized 29 byte log entry
0110 02:51:33.325267 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.325282 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:33.325287 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:33.325348 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.325352 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":129544448,"child_traces":[["op",{"apply.queue_time_us":14383,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":381,"num_ops":2,"prepare.queue_time_us":5519,"prepare.run_cpu_time_us":594,"prepare.run_wall_time_us":898,"replication_time_us":19386}]]}
I20250110 02:51:33.972821 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_2a74c7687def4d02be0bd9827c81eba1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:33.973454 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a74c7687def4d02be0bd9827c81eba1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:33.974457 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:33.974525 1184 tablet_bootstrap.cc:492] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
W20250110 02:51:33.975014 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 36c0aaf2050f481daa8cdac4761fe705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36c0aaf2050f481daa8cdac4761fe705 already in progress: creating tablet
I20250110 02:51:33.975666 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:33.975790 1184 ts_tablet_manager.cc:1397] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20250110 02:51:33.975870 2357 tablet_replica.cc:331] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:33.976061 2357 raft_consensus.cc:2238] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:33.976186 2357 raft_consensus.cc:2267] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.976423 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c0aaf2050f481daa8cdac4761fe705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:33.976389 1184 raft_consensus.cc:357] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.976557 1184 raft_consensus.cc:383] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:33.976604 1184 raft_consensus.cc:738] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:33.977298 2357 ts_tablet_manager.cc:1905] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.977573 2908 raft_consensus.cc:491] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.977680 2908 raft_consensus.cc:513] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.978195 2908 leader_election.cc:290] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.978179 1184 consensus_queue.cc:260] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.979241 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.979218 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd2283e7ed7456dae466a8582587eb7" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.979964 853 leader_election.cc:304] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:33.980165 2378 raft_consensus.cc:2743] T 4bd2283e7ed7456dae466a8582587eb7 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:33.983955 2357 ts_tablet_manager.cc:1918] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:33.984081 2357 log.cc:1198] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/36c0aaf2050f481daa8cdac4761fe705
I20250110 02:51:33.984417 2357 ts_tablet_manager.cc:1939] T 36c0aaf2050f481daa8cdac4761fe705 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:33.985420 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 36c0aaf2050f481daa8cdac4761fe705 (table default.Table_1a66990aee93403b8d03464dec9acd0d [id=c936a0c0b8914e53a4a936795cbd2fdc]) successfully deleted
I20250110 02:51:33.985816 2906 raft_consensus.cc:491] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.985920 2906 raft_consensus.cc:513] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.984092 1184 ts_tablet_manager.cc:1428] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250110 02:51:33.986441 1184 tablet_bootstrap.cc:492] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:33.986452 2906 leader_election.cc:290] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:33.986913 3224 raft_consensus.cc:491] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:33.987032 3224 raft_consensus.cc:513] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:33.987424 2224 raft_consensus.cc:2238] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:33.987540 3224 leader_election.cc:290] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:33.987689 2224 raft_consensus.cc:2267] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:33.987864 1184 tablet_bootstrap.cc:654] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:33.988211 2224 ts_tablet_manager.cc:1905] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:33.988274 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67294cdecffc4ad4b35e000a3e01c9be" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:33.988636 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:33.988811 709 leader_election.cc:343] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:33.989372 994 leader_election.cc:343] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.989854 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:33.990006 774 raft_consensus.cc:2463] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:33.990315 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67294cdecffc4ad4b35e000a3e01c9be" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:33.991487 996 leader_election.cc:304] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:33.991817 2456 raft_consensus.cc:2798] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:33.991937 2456 raft_consensus.cc:491] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:33.992049 2456 raft_consensus.cc:3054] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
W20250110 02:51:33.994868 996 consensus_peers.cc:487] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ae4877bba834dbb96ec9675e2aaedd0. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:33.998176 711 leader_election.cc:343] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:33.998288 711 leader_election.cc:304] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:33.998752 3224 raft_consensus.cc:2743] T 67294cdecffc4ad4b35e000a3e01c9be P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.001646 1184 tablet_bootstrap.cc:492] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.001752 1184 ts_tablet_manager.cc:1397] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250110 02:51:34.002259 1184 raft_consensus.cc:357] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.002465 1184 raft_consensus.cc:383] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.002513 1184 raft_consensus.cc:738] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.003969 1184 consensus_queue.cc:260] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.004962 1184 ts_tablet_manager.cc:1428] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:34.005185 1184 tablet_bootstrap.cc:492] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.005795 3224 raft_consensus.cc:491] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.005892 3224 raft_consensus.cc:513] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.006402 3224 leader_election.cc:290] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.006786 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd238fa1922d456789b59bf5cee52327" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.007346 709 leader_election.cc:343] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.007984 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd238fa1922d456789b59bf5cee52327" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:34.008496 711 leader_election.cc:343] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.008606 711 leader_election.cc:304] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.008812 3224 raft_consensus.cc:2743] T bd238fa1922d456789b59bf5cee52327 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.009696 1184 tablet_bootstrap.cc:654] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.017889 1184 tablet_bootstrap.cc:492] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.018419 1184 ts_tablet_manager.cc:1397] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250110 02:51:34.019239 1184 raft_consensus.cc:357] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.020218 1184 raft_consensus.cc:383] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.020378 1184 raft_consensus.cc:738] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.020654 1184 consensus_queue.cc:260] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.021703 1184 ts_tablet_manager.cc:1428] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250110 02:51:34.021780 3224 raft_consensus.cc:491] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.022465 1184 tablet_bootstrap.cc:492] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.022293 3224 raft_consensus.cc:513] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.023375 3224 leader_election.cc:290] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.023978 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa3527aa074e6ba94960c672aee028" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.024502 709 leader_election.cc:343] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.025388 1184 tablet_bootstrap.cc:654] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.032877 3224 raft_consensus.cc:491] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.033007 3224 raft_consensus.cc:513] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.033702 3224 leader_election.cc:290] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.034511 2378 raft_consensus.cc:491] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.034555 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340882f7684a4e7c9a95800117fc758b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.034636 2378 raft_consensus.cc:513] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:34.034968 709 leader_election.cc:343] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.035137 2378 leader_election.cc:290] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.035610 1184 tablet_bootstrap.cc:492] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.035723 1184 ts_tablet_manager.cc:1397] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250110 02:51:34.036128 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.036264 1184 raft_consensus.cc:357] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.037010 3278 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.037889 1184 raft_consensus.cc:383] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.037962 1184 raft_consensus.cc:738] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.038329 1184 consensus_queue.cc:260] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.039098 1184 ts_tablet_manager.cc:1428] T 7752241af2fb4f959f00252faee87e98 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250110 02:51:34.039352 1184 tablet_bootstrap.cc:492] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.040810 1184 tablet_bootstrap.cc:654] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.045571 2456 raft_consensus.cc:513] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.046190 1184 tablet_bootstrap.cc:492] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.046290 1184 ts_tablet_manager.cc:1397] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250110 02:51:34.046305 2456 leader_election.cc:290] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.046800 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:34.046977 768 raft_consensus.cc:3054] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.048202 1457 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 91f268832c914b88958d6336aa8813af
I20250110 02:51:34.048871 1457 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51672 (request call id 181) took 2643 ms. Trace:
I20250110 02:51:34.049387 768 raft_consensus.cc:2463] T 07c4c653a5954d2789abd1611e3c64d0 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:34.048974 1457 rpcz_store.cc:276] 0110 02:51:31.405340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.227128 (+1821788us) service_pool.cc:225] Handling call
0110 02:51:33.285364 (+ 58236us) spinlock_profiling.cc:243] Waited 58 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfccf6 00007f952feffef6 00007f952feffc25 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.285674 (+ 310us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:33.285777 (+ 103us) catalog_manager.cc:2753] Modifying in-memory table state
0110 02:51:33.285822 (+ 45us) catalog_manager.cc:2762] Locking tablets
0110 02:51:33.285928 (+ 106us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 02:51:33.325781 (+ 39853us) catalog_manager.cc:2794] Removing table from by-name map
0110 02:51:33.325858 (+ 77us) catalog_manager.cc:2810] Committing in-memory state
0110 02:51:34.048132 (+722274us) catalog_manager.cc:2814] Aborting table tasks
0110 02:51:34.048806 (+ 674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.291626 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.292363 (+ 737us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.292368 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:33.292371 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:33.292457 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.292466 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.292470 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:33.292563 (+ 93us) tablet.cc:698] Row locks acquired
0110 02:51:33.292565 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:33.292589 (+ 24us) write_op.cc:270] Start()
0110 02:51:33.292610 (+ 21us) write_op.cc:276] Timestamp: P: 1736477493292551 usec, L: 0
0110 02:51:33.292613 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.292742 (+ 129us) log.cc:844] Serialized 725 byte log entry
0110 02:51:33.310586 (+ 17844us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.325413 (+ 14827us) write_op.cc:301] APPLY: starting
0110 02:51:33.325426 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.325458 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.325481 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.325529 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.325567 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 02:51:33.325574 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:33.325624 (+ 50us) log.cc:844] Serialized 29 byte log entry
0110 02:51:33.325667 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.325679 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:33.325684 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":133378816,"child_traces":[["op",{"apply.queue_time_us":14799,"apply.run_cpu_time_us":314,"apply.run_wall_time_us":314,"num_ops":2,"prepare.queue_time_us":5185,"prepare.run_cpu_time_us":533,"prepare.run_wall_time_us":1403,"replication_time_us":17955}]]}
I20250110 02:51:34.049742 996 leader_election.cc:304] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:34.049873 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_e7476d746857471296ce1afbda5cfa9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:34.046800 1184 raft_consensus.cc:357] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.051057 1184 raft_consensus.cc:383] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.051096 1184 raft_consensus.cc:738] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.051112 2456 raft_consensus.cc:2798] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:34.051213 2456 raft_consensus.cc:695] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:34.051519 2456 consensus_queue.cc:237] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.051658 1184 consensus_queue.cc:260] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.052022 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f268832c914b88958d6336aa8813af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.052066 1184 ts_tablet_manager.cc:1428] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 02:51:34.052548 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f268832c914b88958d6336aa8813af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.053292 1184 tablet_bootstrap.cc:492] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
W20250110 02:51:34.053244 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 91f268832c914b88958d6336aa8813af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91f268832c914b88958d6336aa8813af already in progress: creating tablet
W20250110 02:51:34.053686 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 91f268832c914b88958d6336aa8813af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91f268832c914b88958d6336aa8813af already in progress: creating tablet
W20250110 02:51:34.054059 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7476d746857471296ce1afbda5cfa9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.056217 1184 tablet_bootstrap.cc:654] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.062171 1188 tablet_bootstrap.cc:492] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:34.062474 1188 ts_tablet_manager.cc:1397] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.282s user 0.002s sys 0.001s
I20250110 02:51:34.067248 1184 tablet_bootstrap.cc:492] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.067356 3224 raft_consensus.cc:491] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.067557 1184 ts_tablet_manager.cc:1397] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20250110 02:51:34.067610 3224 raft_consensus.cc:513] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.068084 3224 leader_election.cc:290] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.068658 2224 ts_tablet_manager.cc:1918] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.068662 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1324eeac9446279304f442e2fad488" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.068030 1184 raft_consensus.cc:357] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.068956 1184 raft_consensus.cc:383] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.069015 1184 raft_consensus.cc:738] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.068881 2224 log.cc:1198] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/3ca895eca14748c19fdb7d363499b544
W20250110 02:51:34.069273 709 leader_election.cc:343] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.069619 2224 ts_tablet_manager.cc:1939] T 3ca895eca14748c19fdb7d363499b544 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.070025 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 379) took 3634 ms. Trace:
I20250110 02:51:34.070667 1184 consensus_queue.cc:260] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.071122 1184 ts_tablet_manager.cc:1428] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250110 02:51:34.071363 1184 tablet_bootstrap.cc:492] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.072772 1184 tablet_bootstrap.cc:654] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.076546 3224 raft_consensus.cc:491] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.076663 3224 raft_consensus.cc:513] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.077127 3224 leader_election.cc:290] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.077724 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9abd7180bf64bb5991be5f67ea42fbf" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.078722 709 leader_election.cc:343] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.082747 2378 raft_consensus.cc:491] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.082861 2378 raft_consensus.cc:513] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.083354 2378 leader_election.cc:290] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.083905 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.088179 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskadt4RW/build/asan/bin/kudu pid 424
I20250110 02:51:34.089272 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 02:51:34.091972 2224 rpcz_store.cc:276] 0110 02:51:30.435260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:30.435314 (+ 54us) service_pool.cc:225] Handling call
0110 02:51:33.952023 (+3516709us) ts_tablet_manager.cc:1145] Deleting tablet 3ca895eca14748c19fdb7d363499b544
0110 02:51:33.952032 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:33.991315 (+ 39283us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.068558 (+ 77243us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.070008 (+ 1450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3477855}
I20250110 02:51:34.092458 2224 tablet_replica.cc:331] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.092866 2224 raft_consensus.cc:2238] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.093084 2224 raft_consensus.cc:2267] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.093688 2224 ts_tablet_manager.cc:1905] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.099063 1188 raft_consensus.cc:357] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.099359 1188 raft_consensus.cc:383] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.099499 1188 raft_consensus.cc:738] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.099960 1188 consensus_queue.cc:260] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.128216 3283 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.135213 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 3ca895eca14748c19fdb7d363499b544 (table default.Table_bb82c27076c5492e9788b3e1b4a66c1a [id=0431f94b7f3740b1ad7cc9fea1b4eb47]) successfully deleted
I20250110 02:51:34.140810 3294 tablet.cc:1721] T 235ffe7bf29a4cbcb4faef6f3b2931ec P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.141985 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c4c653a5954d2789abd1611e3c64d0" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:34.143222 3224 raft_consensus.cc:491] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.143510 3224 raft_consensus.cc:513] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.144050 3224 leader_election.cc:290] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.144672 917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1f2acd39be40e4b71cc0dcf68e7567" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.145152 709 leader_election.cc:343] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.146188 2456 raft_consensus.cc:491] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.146346 2456 raft_consensus.cc:513] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.147116 2456 leader_election.cc:290] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.147497 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.147689 772 raft_consensus.cc:2463] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:34.149797 3276 raft_consensus.cc:491] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.149984 3276 raft_consensus.cc:513] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.150774 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.151473 771 raft_consensus.cc:2463] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 0.
I20250110 02:51:34.151615 996 leader_election.cc:304] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:34.151995 996 leader_election.cc:304] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
I20250110 02:51:34.152341 3281 raft_consensus.cc:2798] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:34.152410 3281 raft_consensus.cc:491] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:34.152501 3281 raft_consensus.cc:3054] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.153666 2906 raft_consensus.cc:2798] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:34.153731 2906 raft_consensus.cc:491] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:34.153790 2906 raft_consensus.cc:3054] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.154732 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadb829d657346eabcb7d2aa34b33fe2" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.154846 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f268832c914b88958d6336aa8813af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:33 UTC) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:34.155539 917 raft_consensus.cc:3054] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.155582 2224 ts_tablet_manager.cc:1918] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.155663 2224 log.cc:1198] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/0558a2c03f6b42a6befdc48e3af23a6d
I20250110 02:51:34.155964 2224 ts_tablet_manager.cc:1939] T 0558a2c03f6b42a6befdc48e3af23a6d P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.169045 3276 leader_election.cc:290] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:34.175647 1181 tablet_bootstrap.cc:492] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.175760 1181 ts_tablet_manager.cc:1397] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.363s user 0.009s sys 0.025s
I20250110 02:51:34.176350 1181 raft_consensus.cc:357] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.176507 1181 raft_consensus.cc:383] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.176559 1181 raft_consensus.cc:738] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.176761 1181 consensus_queue.cc:260] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.177101 1181 ts_tablet_manager.cc:1428] T 4d8f8d0ac49a428fb7cbabb1e27bf34e P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:34.177615 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 396) took 2986 ms. Trace:
I20250110 02:51:34.177680 2224 rpcz_store.cc:276] 0110 02:51:31.190757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.193158 (+ 2401us) service_pool.cc:225] Handling call
0110 02:51:34.092402 (+2899244us) ts_tablet_manager.cc:1145] Deleting tablet 0558a2c03f6b42a6befdc48e3af23a6d
0110 02:51:34.092415 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.135754 (+ 43339us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.155500 (+ 19746us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.177598 (+ 22098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899049}
I20250110 02:51:34.179001 1181 tablet_bootstrap.cc:492] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.180681 1181 tablet_bootstrap.cc:654] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.182999 2224 tablet_replica.cc:331] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.183465 2224 raft_consensus.cc:2238] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:34.183732 2224 pending_rounds.cc:62] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:34.183799 2224 pending_rounds.cc:69] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611810279055360 op_type: NO_OP noop_request { }
I20250110 02:51:34.183900 2224 raft_consensus.cc:2883] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:34.183995 2224 raft_consensus.cc:2267] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.184438 2224 ts_tablet_manager.cc:1905] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.190469 1439 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: default.Table_b127feb5603541859f25e8a7b3703c7d [id=510ab81389564eb4af6f119bb0e6f701] alter complete (version 1)
I20250110 02:51:34.196864 2224 ts_tablet_manager.cc:1918] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.196965 2224 log.cc:1198] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/eed058931194480ca1a27130bfa8ee32
I20250110 02:51:34.197273 2224 ts_tablet_manager.cc:1939] T eed058931194480ca1a27130bfa8ee32 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.197306 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.197863 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 397) took 2990 ms. Trace:
W20250110 02:51:34.197935 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 4de67338a26f4febb5dd7611a72b4723 because the tablet was not found. No further retry: Not found: Tablet not found: 4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:34.197937 2224 rpcz_store.cc:276] 0110 02:51:31.207027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.208998 (+ 1971us) service_pool.cc:225] Handling call
0110 02:51:34.177882 (+2968884us) ts_tablet_manager.cc:1145] Deleting tablet eed058931194480ca1a27130bfa8ee32
0110 02:51:34.182940 (+ 5058us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.194219 (+ 11279us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.196797 (+ 2578us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.197846 (+ 1049us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2968710}
I20250110 02:51:34.199218 1181 tablet_bootstrap.cc:492] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.199440 1181 ts_tablet_manager.cc:1397] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250110 02:51:34.200317 1181 raft_consensus.cc:357] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.200590 1181 raft_consensus.cc:383] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.202435 1181 raft_consensus.cc:738] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.202762 1181 consensus_queue.cc:260] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.202821 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27f7ffc50494854ba34e1fbce9bee1b" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.203305 1181 ts_tablet_manager.cc:1428] T 07acc4681ae94a069003ed75fcce5141 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:34.203476 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 398) took 2996 ms. Trace:
I20250110 02:51:34.203562 1181 tablet_bootstrap.cc:492] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.204768 1441 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 183) took 2751 ms. Trace:
I20250110 02:51:34.204838 1181 tablet_bootstrap.cc:654] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.204926 1441 rpcz_store.cc:276] 0110 02:51:31.452859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.268919 (+1816060us) service_pool.cc:225] Handling call
0110 02:51:33.287681 (+ 18762us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.288584 (+ 903us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:33.288813 (+ 229us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:33.288955 (+ 142us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:33.289027 (+ 72us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:33.329620 (+ 40593us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:34.204716 (+875096us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.293784 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.293848 (+ 64us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.293852 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:33.293854 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.293978 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.293987 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.293991 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:33.294025 (+ 34us) tablet.cc:698] Row locks acquired
0110 02:51:33.294027 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:33.294048 (+ 21us) write_op.cc:270] Start()
0110 02:51:33.294069 (+ 21us) write_op.cc:276] Timestamp: P: 1736477493294011 usec, L: 0
0110 02:51:33.294073 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.294176 (+ 103us) log.cc:844] Serialized 405 byte log entry
0110 02:51:33.326224 (+ 32048us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.326623 (+ 399us) write_op.cc:301] APPLY: starting
0110 02:51:33.326631 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.326654 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.326678 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.326706 (+ 28us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.326726 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:33.326732 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:33.326775 (+ 43us) log.cc:844] Serialized 23 byte log entry
0110 02:51:33.326812 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.326823 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:33.326827 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":42549120,"child_traces":[["op",{"apply.queue_time_us":371,"apply.run_cpu_time_us":243,"apply.run_wall_time_us":242,"num_ops":1,"prepare.queue_time_us":4392,"prepare.run_cpu_time_us":425,"prepare.run_wall_time_us":2152,"replication_time_us":32133}]]}
I20250110 02:51:34.209432 1440 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51672 (request call id 180) took 2804 ms. Trace:
I20250110 02:51:34.209661 2224 rpcz_store.cc:276] 0110 02:51:31.207192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.209212 (+ 2020us) service_pool.cc:225] Handling call
0110 02:51:34.198139 (+2988927us) ts_tablet_manager.cc:1145] Deleting tablet e4726347afd649afa942d74df8c013fe
0110 02:51:34.198147 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.203463 (+ 5316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2988786}
I20250110 02:51:34.210002 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 400) took 2846 ms. Trace:
I20250110 02:51:34.210070 2224 rpcz_store.cc:276] 0110 02:51:31.363065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.363104 (+ 39us) service_pool.cc:225] Handling call
0110 02:51:34.209892 (+2846788us) ts_tablet_manager.cc:1145] Deleting tablet 4f015e99f53f48599ac2d7a29220bc39
0110 02:51:34.209900 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.209990 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846637}
I20250110 02:51:34.209592 1440 rpcz_store.cc:276] 0110 02:51:31.404812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.213803 (+1808991us) service_pool.cc:225] Handling call
0110 02:51:33.286576 (+ 72773us) spinlock_profiling.cc:243] Waited 72.3 ms on lock 0x616000051454. stack: 00007f95258f2fd6 00007f95258f2d90 00007f95248d248d 000000000096d544 000000000096d50d 00007f9534490e1d 00007f9534490f13 00007f9531199211 00007f9538c709dd 00007f9538dfdf96 00007f952ff01bb6 00007f952ff018e5 00007f952c65a30e 00007f952c6591ee 00007f952c65f425 00007f952c65fd4f
0110 02:51:33.286815 (+ 239us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 02:51:33.287006 (+ 191us) catalog_manager.cc:3627] Apply alter schema
0110 02:51:33.287176 (+ 170us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 02:51:33.287235 (+ 59us) catalog_manager.cc:3817] Updating metadata on disk
0110 02:51:33.326578 (+ 39343us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 02:51:34.209396 (+882818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.293070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.293145 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.293149 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:33.293151 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.293210 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.293216 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.293220 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:33.293283 (+ 63us) tablet.cc:698] Row locks acquired
0110 02:51:33.293286 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:33.293307 (+ 21us) write_op.cc:270] Start()
0110 02:51:33.293339 (+ 32us) write_op.cc:276] Timestamp: P: 1736477493293270 usec, L: 0
0110 02:51:33.293342 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.293452 (+ 110us) log.cc:844] Serialized 405 byte log entry
0110 02:51:33.325968 (+ 32516us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.326276 (+ 308us) write_op.cc:301] APPLY: starting
0110 02:51:33.326284 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.326307 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.326333 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.326363 (+ 30us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.326402 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:33.326409 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:33.326453 (+ 44us) log.cc:844] Serialized 23 byte log entry
0110 02:51:33.326496 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.326507 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:33.326513 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:33.326572 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.326575 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":166290304,"child_traces":[["op",{"apply.queue_time_us":276,"apply.run_cpu_time_us":288,"apply.run_wall_time_us":287,"num_ops":1,"prepare.queue_time_us":5509,"prepare.run_cpu_time_us":477,"prepare.run_wall_time_us":708,"raft.queue_time_us":29,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":131,"replication_time_us":32605,"spinlock_wait_cycles":260864}]]}
I20250110 02:51:34.210281 2224 tablet_replica.cc:331] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.210449 2224 raft_consensus.cc:2238] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:34.210670 2224 raft_consensus.cc:2267] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.211155 2224 ts_tablet_manager.cc:1905] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.211469 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_889050ea4d1a42e6b9521793c071f5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.212074 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_889050ea4d1a42e6b9521793c071f5e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.217849 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa3527aa074e6ba94960c672aee028" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.218396 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340882f7684a4e7c9a95800117fc758b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.219113 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1324eeac9446279304f442e2fad488" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.219530 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9abd7180bf64bb5991be5f67ea42fbf" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.219931 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1f2acd39be40e4b71cc0dcf68e7567" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.220353 853 leader_election.cc:304] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:34.220824 2378 raft_consensus.cc:2743] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.101037 1188 ts_tablet_manager.cc:1428] T ac5dab241cd74f329a57e6f19e95a7b6 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20250110 02:51:34.224485 1188 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:34.225993 1188 tablet_bootstrap.cc:654] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.229920 3281 raft_consensus.cc:513] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.221212 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet 0558a2c03f6b42a6befdc48e3af23a6d (table default.Table_809c92784a3546a4b5a3a6fd7d7a9e4c [id=08e87fcaa10c4d8f9adb9468bea8980e]) successfully deleted
I20250110 02:51:34.232695 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet eed058931194480ca1a27130bfa8ee32 (table default.Table_26326c3a6b1b404bb42e75c5a619be95 [id=78b013e77cf340089467193ef7c17819]) successfully deleted
W20250110 02:51:34.245523 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e4726347afd649afa942d74df8c013fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4726347afd649afa942d74df8c013fe already in progress: creating tablet
W20250110 02:51:34.245992 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4f015e99f53f48599ac2d7a29220bc39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f015e99f53f48599ac2d7a29220bc39 already in progress: creating tablet
W20250110 02:51:34.246469 711 leader_election.cc:343] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.246599 711 leader_election.cc:304] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.247046 3224 raft_consensus.cc:2743] T deaa3527aa074e6ba94960c672aee028 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.247416 711 leader_election.cc:343] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.247514 711 leader_election.cc:304] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.247890 3224 raft_consensus.cc:2743] T 340882f7684a4e7c9a95800117fc758b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.250741 711 leader_election.cc:343] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.250862 711 leader_election.cc:304] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.251080 3224 raft_consensus.cc:2743] T fc1324eeac9446279304f442e2fad488 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.251263 711 leader_election.cc:343] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.251366 711 leader_election.cc:304] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.251561 3224 raft_consensus.cc:2743] T e9abd7180bf64bb5991be5f67ea42fbf P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.251713 711 leader_election.cc:343] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.251811 711 leader_election.cc:304] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.252718 3224 raft_consensus.cc:2743] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.253971 3224 raft_consensus.cc:491] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.254070 3224 raft_consensus.cc:513] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.254684 3224 leader_election.cc:290] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.274709 3224 raft_consensus.cc:491] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.274845 3224 raft_consensus.cc:513] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.275782 3224 leader_election.cc:290] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.276621 3283 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 98336b684b41407da8a1949e30cd82e0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.241068 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.241498 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.243041 915 raft_consensus.cc:3054] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.245184 853 leader_election.cc:304] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:34.255156 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2b5a305db34fedbbbf0d666e6c28cb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.275717 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8600a637306741dd8e1e7f4964ff4bc8" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.285861 709 leader_election.cc:343] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:34.286053 709 leader_election.cc:343] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.293038 2908 raft_consensus.cc:491] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.293171 2908 raft_consensus.cc:513] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.293728 2908 leader_election.cc:290] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:34.294700 853 leader_election.cc:343] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: 213939cf13354ce380fb7cda5b38676f
I20250110 02:51:34.295492 2378 raft_consensus.cc:2743] T e27f7ffc50494854ba34e1fbce9bee1b P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.298563 3224 raft_consensus.cc:491] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.298753 3224 raft_consensus.cc:513] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.299587 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.300122 3224 leader_election.cc:290] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:34.300280 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 3ca895eca14748c19fdb7d363499b544 because the tablet was not found. No further retry: Not found: Tablet not found: 3ca895eca14748c19fdb7d363499b544
I20250110 02:51:34.300931 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.301088 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b937188bf2949af8f54bade4e0ad984 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e53b838be846c7b3ac44dc75aa3730) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.301211 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 071bbce1706c43b6bb690c7de1ead3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.301321 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
W20250110 02:51:34.301285 1448 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 9e28d8769f364e10aad54b60fb221f35 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9e28d8769f364e10aad54b60fb221f35 consensus state has no leader
I20250110 02:51:34.301434 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f1782c46a44ed091f4c533226c71f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:22 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.301538 740 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.301638 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8444c1381a45d6b2fee30da971bd47" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.301745 2357 tablet_replica.cc:331] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:34.301921 2357 raft_consensus.cc:2238] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:34.301981 709 leader_election.cc:343] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.302063 2357 raft_consensus.cc:2267] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.304560 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e4726347afd649afa942d74df8c013fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.304709 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fe860d60b3194dc5a2e7a94d39d4b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 471862cae972410b9770f7a8742cac0a) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.304801 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de67338a26f4febb5dd7611a72b4723 with delete_type TABLET_DATA_DELETED (replaced by tablet 68f13fd7b6eb4c6b97b86cb9b4b00cac) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.304904 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f015e99f53f48599ac2d7a29220bc39 with delete_type TABLET_DATA_DELETED (replaced by tablet 81138947fcab4db69bf1492de1f8560f) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.304998 740 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305101 742 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8444c1381a45d6b2fee30da971bd47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:20 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305210 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d634dd0c2044f91b304fe3737330d88 with delete_type TABLET_DATA_DELETED (replaced by tablet 587e4962bcc44e7ebcce808beb2a8acf) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305291 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f69d0c201c524b7e95c9d3851b8dd8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305368 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b45b91d084a44c1e8cdf1553967878e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305444 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8527249404ac4be01021778e64fbe) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305581 743 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6b2654f86e4686b8a3034de02d11bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305693 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e55ee565b7f14936a2cb4cbed716c153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305809 740 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1324eeac9446279304f442e2fad488 with delete_type TABLET_DATA_DELETED (replaced by tablet fd02e9f8cd0748a0a6a3202fbe390ad1) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305897 742 tablet_service.cc:1514] Processing DeleteTablet for tablet e848586bec574a529caaf5e498db79cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.305986 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d064a7e8ea34f18b42407903daf7197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306072 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0f754966c04074a568c05063591990 with delete_type TABLET_DATA_DELETED (replaced by tablet c1885ab76b5c424d82fddd6b5481e4d6) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306174 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d6acae633eb14e5db87b1e36a069422c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306298 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 213939cf13354ce380fb7cda5b38676f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306423 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae4877bba834dbb96ec9675e2aaedd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306524 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f9b3e74ee2431a85a22fefaacfa40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:27 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306622 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 24034fb3da6e47978cdc9b02e714e30d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:24 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.306712 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6598311f59c6476ba339b8e57961cf67 with delete_type TABLET_DATA_DELETED (replaced by tablet 2312c1ee4a134caaa68980a224dd283e) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.307181 2357 ts_tablet_manager.cc:1905] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:34.309504 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 7c2c7a313ba94e5ea85e45276658ba04 because the tablet was not found. No further retry: Not found: Tablet not found: 7c2c7a313ba94e5ea85e45276658ba04
W20250110 02:51:34.309682 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 9b937188bf2949af8f54bade4e0ad984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b937188bf2949af8f54bade4e0ad984 already in progress: creating tablet
W20250110 02:51:34.309821 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 071bbce1706c43b6bb690c7de1ead3fe because the tablet was not found. No further retry: Not found: Tablet not found: 071bbce1706c43b6bb690c7de1ead3fe
I20250110 02:51:34.311079 740 tablet_service.cc:1514] Processing DeleteTablet for tablet eed058931194480ca1a27130bfa8ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.311223 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558a2c03f6b42a6befdc48e3af23a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.311748 3281 leader_election.cc:290] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.234457 2906 raft_consensus.cc:513] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.249392 2224 ts_tablet_manager.cc:1918] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.312127 2224 log.cc:1198] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e804fe33c8ae489bbc4d3765c64b59bd
I20250110 02:51:34.312108 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:34.312260 772 raft_consensus.cc:3054] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.312454 2224 ts_tablet_manager.cc:1939] T e804fe33c8ae489bbc4d3765c64b59bd P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.312469 2906 leader_election.cc:290] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Requested vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:34.313155 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 407) took 2740 ms. Trace:
I20250110 02:51:34.313220 2224 rpcz_store.cc:276] 0110 02:51:31.572621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.572669 (+ 48us) service_pool.cc:225] Handling call
0110 02:51:34.210249 (+2637580us) ts_tablet_manager.cc:1145] Deleting tablet e804fe33c8ae489bbc4d3765c64b59bd
0110 02:51:34.210256 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.241207 (+ 30951us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.249295 (+ 8088us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.313138 (+ 63843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2637401}
I20250110 02:51:34.313436 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f0d4d6e02b1d4334a9993d320217f1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:28 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.313524 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 408) took 2499 ms. Trace:
I20250110 02:51:34.313582 2224 rpcz_store.cc:276] 0110 02:51:31.814054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:31.814106 (+ 52us) service_pool.cc:225] Handling call
0110 02:51:34.313434 (+2499328us) ts_tablet_manager.cc:1145] Deleting tablet 6598311f59c6476ba339b8e57961cf67
0110 02:51:34.313442 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.313512 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2473311}
I20250110 02:51:34.313791 2224 tablet_replica.cc:331] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.313943 2224 raft_consensus.cc:2238] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:34.313948 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet f0d4d6e02b1d4334a9993d320217f1f8 because the tablet was not found. No further retry: Not found: Tablet not found: f0d4d6e02b1d4334a9993d320217f1f8
I20250110 02:51:34.314082 2224 raft_consensus.cc:2267] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.273332 3278 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 8899dd1aeb384e4ea4523f5fd225c888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.314527 772 raft_consensus.cc:2463] T b48fe2d061da4e729c8704af4fc83f7a P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:34.314818 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c7a313ba94e5ea85e45276658ba04 with delete_type TABLET_DATA_DELETED (replaced by tablet 30645a0642bc43ff957368f928b877d1) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.315150 996 leader_election.cc:304] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:34.315244 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 7c2c7a313ba94e5ea85e45276658ba04 because the tablet was not found. No further retry: Not found: Tablet not found: 7c2c7a313ba94e5ea85e45276658ba04
I20250110 02:51:34.315420 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0"
I20250110 02:51:34.315554 772 raft_consensus.cc:3054] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.315788 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e05a7554a14805bdcbd1824a5bd9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:15 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:34.316200 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 04e05a7554a14805bdcbd1824a5bd9d5 because the tablet was not found. No further retry: Not found: Tablet not found: 04e05a7554a14805bdcbd1824a5bd9d5
I20250110 02:51:34.316865 879 tablet_service.cc:1514] Processing DeleteTablet for tablet c00e97e496b54dcf82b7182591d6f4aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:34.317298 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet c00e97e496b54dcf82b7182591d6f4aa because the tablet was not found. No further retry: Not found: Tablet not found: c00e97e496b54dcf82b7182591d6f4aa
I20250110 02:51:34.317637 772 raft_consensus.cc:2463] T 1df15c10ef2d4bf99af06a860c6f250f P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8899dd1aeb384e4ea4523f5fd225c888 in term 1.
I20250110 02:51:34.317873 879 tablet_service.cc:1514] Processing DeleteTablet for tablet bb595013e809480a814cb9c7aed856e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e28d8769f364e10aad54b60fb221f35) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.317991 996 leader_election.cc:304] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters:
W20250110 02:51:34.318341 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet bb595013e809480a814cb9c7aed856e6 because the tablet was not found. No further retry: Not found: Tablet not found: bb595013e809480a814cb9c7aed856e6
I20250110 02:51:34.318437 3281 raft_consensus.cc:2798] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:34.318545 3281 raft_consensus.cc:695] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:34.318501 2456 raft_consensus.cc:2798] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:34.318740 2456 raft_consensus.cc:695] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Becoming Leader. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Running, Role: LEADER
I20250110 02:51:34.318796 3281 consensus_queue.cc:237] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.318979 2456 consensus_queue.cc:237] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
W20250110 02:51:34.279882 994 leader_election.cc:343] T 07c4c653a5954d2789abd1611e3c64d0 P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.280573 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2b5a305db34fedbbbf0d666e6c28cb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.320130 1053 raft_consensus.cc:2463] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 0.
I20250110 02:51:34.310051 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8600a637306741dd8e1e7f4964ff4bc8" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.310231 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8444c1381a45d6b2fee30da971bd47" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.310202 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213939cf13354ce380fb7cda5b38676f" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:34.320900 994 consensus_peers.cc:487] T 7c55788eeddb47aa8a5893703c4d5e67 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:34.321424 994 consensus_peers.cc:487] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:34.321761 2224 ts_tablet_manager.cc:1905] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:34.323071 994 consensus_peers.cc:487] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:34.323199 994 leader_election.cc:343] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:34.323424 994 leader_election.cc:343] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.325208 2378 raft_consensus.cc:491] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.325347 2378 raft_consensus.cc:513] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.325870 2378 leader_election.cc:290] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
I20250110 02:51:34.326035 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb805b3e537466ab7fe0185421ad5aa with delete_type TABLET_DATA_DELETED (replaced by tablet d48a22b9642a442e83a3ecb7117d5c80) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.326241 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa78a9ef5eb443da0c8983a7190440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:21 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.326359 743 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.326506 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e804fe33c8ae489bbc4d3765c64b59bd with delete_type TABLET_DATA_DELETED (replaced by tablet 2fd9b07cb5604fed8b3eb1b576353940) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.326618 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a0ddab9e847445999ec97eac4352d9c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e26341525d462e9042b1ab06cf7ac3) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.326748 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b379b357194d729108428cbaebd51a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.326916 2352 tablet_replica.cc:331] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
W20250110 02:51:34.326978 1448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:57009 (request call id 134) took 1854 ms (client timeout 1000 ms). Trace:
I20250110 02:51:34.327088 2352 raft_consensus.cc:2238] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.327225 2352 raft_consensus.cc:2267] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.327709 2352 ts_tablet_manager.cc:1905] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:34.327067 1448 rpcz_store.cc:269] 0110 02:51:32.472235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:32.710249 (+238014us) service_pool.cc:225] Handling call
0110 02:51:34.326931 (+1616682us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:32.949729 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:32.949839 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:32.949843 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:32.949845 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:32.949936 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:32.949944 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:32.949948 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:32.950030 (+ 82us) tablet.cc:698] Row locks acquired
0110 02:51:32.950033 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:32.950067 (+ 34us) write_op.cc:270] Start()
0110 02:51:32.950100 (+ 33us) write_op.cc:276] Timestamp: P: 1736477492950003 usec, L: 0
0110 02:51:32.950105 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:32.950261 (+ 156us) log.cc:844] Serialized 527 byte log entry
0110 02:51:32.990551 (+ 40290us) op_driver.cc:464] REPLICATION: finished
0110 02:51:32.991374 (+ 823us) write_op.cc:301] APPLY: starting
0110 02:51:32.991385 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:32.991415 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:32.991456 (+ 41us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:32.991499 (+ 43us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:32.991526 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:32.991533 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:32.991618 (+ 85us) log.cc:844] Serialized 23 byte log entry
0110 02:51:32.991656 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:32.991673 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:32.991679 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:33.175791 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.175866 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.175870 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:33.175872 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.175964 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.176001 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.176260 (+ 259us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:33.176320 (+ 60us) tablet.cc:698] Row locks acquired
0110 02:51:33.176322 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:33.176349 (+ 27us) write_op.cc:270] Start()
0110 02:51:33.176370 (+ 21us) write_op.cc:276] Timestamp: P: 1736477493176321 usec, L: 0
0110 02:51:33.176373 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.176488 (+ 115us) log.cc:844] Serialized 356 byte log entry
0110 02:51:33.196382 (+ 19894us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.197068 (+ 686us) write_op.cc:301] APPLY: starting
0110 02:51:33.197079 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.197105 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.197132 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.197177 (+ 45us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.197200 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:33.197207 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:33.197274 (+ 67us) log.cc:844] Serialized 23 byte log entry
0110 02:51:33.197327 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.197340 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:33.197345 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":5956,"reported_tablets":240,"child_traces":[["op",{"apply.queue_time_us":788,"apply.run_cpu_time_us":364,"apply.run_wall_time_us":364,"num_ops":1,"prepare.queue_time_us":636,"prepare.run_cpu_time_us":7442,"prepare.run_wall_time_us":151358,"raft.queue_time_us":4788,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":40431,"thread_start_us":289915,"threads_started":2}],["op",{"apply.queue_time_us":650,"apply.run_cpu_time_us":341,"apply.run_wall_time_us":342,"num_ops":1,"prepare.queue_time_us":150165,"prepare.run_cpu_time_us":504,"prepare.run_wall_time_us":741,"replication_time_us":19995,"thread_start_us":80955,"threads_started":1}]]}
I20250110 02:51:34.331002 1188 tablet_bootstrap.cc:492] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:34.331136 1188 ts_tablet_manager.cc:1397] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.107s user 0.001s sys 0.002s
I20250110 02:51:34.331777 1188 raft_consensus.cc:357] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.332617 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.333473 1188 raft_consensus.cc:383] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.333596 1188 raft_consensus.cc:738] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.333634 2352 ts_tablet_manager.cc:1918] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.333799 2352 log.cc:1198] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/f9b379b357194d729108428cbaebd51a
I20250110 02:51:34.333904 1188 consensus_queue.cc:260] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.334154 2352 ts_tablet_manager.cc:1939] T f9b379b357194d729108428cbaebd51a P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:34.334503 1188 ts_tablet_manager.cc:1428] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 02:51:34.335219 879 tablet_service.cc:1514] Processing DeleteTablet for tablet dba0cb7970f1453dac256b255ed7cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:17 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.335844 892 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb726d2c9a04963b92a3fb625cf84f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.336525 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fe86be26d1b544c08b231f20085d3159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:31 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.337061 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ca895eca14748c19fdb7d363499b544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:29 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.338150 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 80186dd7d1a240068584b64ce9e9b03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:14 UTC) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.339445 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d51adff7f50c4aec94e1892a50ec36c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 02:51:13 UTC) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:34.340145 1439 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:47877 (request call id 115) took 1270 ms (client timeout 1000 ms). Trace:
I20250110 02:51:34.345515 2357 ts_tablet_manager.cc:1918] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:34.345670 2357 log.cc:1198] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/8d1f2acd39be40e4b71cc0dcf68e7567
I20250110 02:51:34.346441 2357 ts_tablet_manager.cc:1939] T 8d1f2acd39be40e4b71cc0dcf68e7567 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:34.347599 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 8d1f2acd39be40e4b71cc0dcf68e7567 (table default.Table_9cd2f808ffbc4122be3501fc1a25c768 [id=27b4ab40c050453f8af5d9c8b63523e9]) successfully deleted
I20250110 02:51:34.348021 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 186) took 2826 ms. Trace:
I20250110 02:51:34.348183 1457 rpcz_store.cc:276] 0110 02:51:31.522032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.049621 (+2527589us) service_pool.cc:225] Handling call
0110 02:51:34.212205 (+162584us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.212607 (+ 402us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.249461 (+ 36854us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.347883 (+ 98422us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.347967 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.238902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.238989 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.238998 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:34.239000 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.239073 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.239082 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.239087 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.239248 (+ 161us) tablet.cc:698] Row locks acquired
0110 02:51:34.239252 (+ 4us) write_op.cc:260] PREPARE: finished
0110 02:51:34.239282 (+ 30us) write_op.cc:270] Start()
0110 02:51:34.239310 (+ 28us) write_op.cc:276] Timestamp: P: 1736477494239250 usec, L: 0
0110 02:51:34.239315 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.239453 (+ 138us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.248578 (+ 9125us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.248966 (+ 388us) write_op.cc:301] APPLY: starting
0110 02:51:34.248979 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.249012 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.249043 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.249173 (+ 130us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.249219 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.249231 (+ 12us) write_op.cc:312] APPLY: finished
0110 02:51:34.249295 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.249356 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.249369 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:34.249374 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.249459 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.249463 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":338,"apply.run_cpu_time_us":471,"apply.run_wall_time_us":472,"num_ops":2,"prepare.queue_time_us":25871,"prepare.run_cpu_time_us":556,"prepare.run_wall_time_us":612,"replication_time_us":9240}]]}
I20250110 02:51:34.349047 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48fe2d061da4e729c8704af4fc83f7a" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:34.351881 1188 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:34.351099 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_ca02b975db9740d3b778e3ceb5255486"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.352483 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca02b975db9740d3b778e3ceb5255486 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.352874 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet f9b379b357194d729108428cbaebd51a (table default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]) successfully deleted
I20250110 02:51:34.352942 2352 tablet_replica.cc:331] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:34.353204 2352 raft_consensus.cc:2238] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.353438 2352 raft_consensus.cc:2267] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.354194 1188 tablet_bootstrap.cc:654] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.354295 2352 ts_tablet_manager.cc:1905] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.357844 2224 ts_tablet_manager.cc:1918] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250110 02:51:34.358033 2224 log.cc:1198] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/fe86be26d1b544c08b231f20085d3159
I20250110 02:51:34.358490 2224 ts_tablet_manager.cc:1939] T fe86be26d1b544c08b231f20085d3159 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.359303 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 425) took 1655 ms. Trace:
I20250110 02:51:34.359434 2224 rpcz_store.cc:276] 0110 02:51:32.703848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:32.704170 (+ 322us) service_pool.cc:225] Handling call
0110 02:51:34.313757 (+1609587us) ts_tablet_manager.cc:1145] Deleting tablet fe86be26d1b544c08b231f20085d3159
0110 02:51:34.313765 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.336841 (+ 23076us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.357734 (+ 20893us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.359284 (+ 1550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1609444}
I20250110 02:51:34.359920 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 426) took 1643 ms. Trace:
I20250110 02:51:34.360246 2224 rpcz_store.cc:276] 0110 02:51:32.716300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:32.716357 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:34.359788 (+1643431us) ts_tablet_manager.cc:1145] Deleting tablet e55ee565b7f14936a2cb4cbed716c153
0110 02:51:34.359797 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.359900 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621189}
I20250110 02:51:34.360644 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 429) took 1361 ms. Trace:
I20250110 02:51:34.362548 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 185) took 2893 ms. Trace:
I20250110 02:51:34.362689 1450 rpcz_store.cc:276] 0110 02:51:31.468892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.972526 (+2503634us) service_pool.cc:225] Handling call
0110 02:51:34.213043 (+240517us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.213419 (+ 376us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.250017 (+ 36598us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.362368 (+112351us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.362484 (+ 116us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.239557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.239645 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.239650 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:34.239652 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.239716 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.239723 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.239727 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.239797 (+ 70us) tablet.cc:698] Row locks acquired
0110 02:51:34.239799 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.239823 (+ 24us) write_op.cc:270] Start()
0110 02:51:34.239847 (+ 24us) write_op.cc:276] Timestamp: P: 1736477494239792 usec, L: 0
0110 02:51:34.239852 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.239982 (+ 130us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.248626 (+ 8644us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.249536 (+ 910us) write_op.cc:301] APPLY: starting
0110 02:51:34.249550 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.249588 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.249616 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.249746 (+ 130us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.249782 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.249792 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:34.249856 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.249928 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.249945 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:34.249950 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":875,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":486,"num_ops":2,"prepare.queue_time_us":25685,"prepare.run_cpu_time_us":495,"prepare.run_wall_time_us":482,"replication_time_us":8761,"thread_start_us":75863,"threads_started":1}]]}
I20250110 02:51:34.363823 1440 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 187) took 2787 ms. Trace:
I20250110 02:51:34.364131 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_5f2623d7fcb5415c8468e4aa77691a5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:34.363952 1440 rpcz_store.cc:276] 0110 02:51:31.575740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.211277 (+2635537us) service_pool.cc:225] Handling call
0110 02:51:34.346779 (+135502us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.347330 (+ 551us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.363393 (+ 16063us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.363529 (+ 136us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.363612 (+ 83us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.352664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.352752 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.352761 (+ 9us) write_op.cc:435] Acquired schema lock
0110 02:51:34.353116 (+ 355us) tablet.cc:659] Decoding operations
0110 02:51:34.353236 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.353249 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.353254 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.353331 (+ 77us) tablet.cc:698] Row locks acquired
0110 02:51:34.353333 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.353365 (+ 32us) write_op.cc:270] Start()
0110 02:51:34.353397 (+ 32us) write_op.cc:276] Timestamp: P: 1736477494353321 usec, L: 0
0110 02:51:34.353400 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.353564 (+ 164us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.361297 (+ 7733us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.361521 (+ 224us) write_op.cc:301] APPLY: starting
0110 02:51:34.361539 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.361589 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.361627 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.361784 (+ 157us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.361827 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.361837 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:34.363104 (+ 1267us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.363189 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.363217 (+ 28us) write_op.cc:454] Released schema lock
0110 02:51:34.363224 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.363299 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.363305 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":592,"apply.run_wall_time_us":1770,"num_ops":2,"prepare.queue_time_us":4837,"prepare.run_cpu_time_us":677,"prepare.run_wall_time_us":9660,"raft.queue_time_us":37,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":241,"replication_time_us":7864}]]}
I20250110 02:51:34.364985 1440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_5b5d73d6f46a438383b6a8439d93c37e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.365235 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f2623d7fcb5415c8468e4aa77691a5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:34.365590 1440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b5d73d6f46a438383b6a8439d93c37e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.366509 912 raft_consensus.cc:3054] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.369647 3224 consensus_queue.cc:1035] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:34.371557 1188 tablet_bootstrap.cc:492] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:34.371727 1188 ts_tablet_manager.cc:1397] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20250110 02:51:34.372491 1188 raft_consensus.cc:357] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.372750 1188 raft_consensus.cc:383] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.372857 1188 raft_consensus.cc:738] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.373142 1188 consensus_queue.cc:260] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.373783 1188 ts_tablet_manager.cc:1428] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:34.374060 1188 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:34.375558 1188 tablet_bootstrap.cc:654] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:34.376749 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:34.385191 1440 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 190) took 2744 ms. Trace:
I20250110 02:51:34.385324 1440 rpcz_store.cc:276] 0110 02:51:31.641151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.364735 (+2723584us) service_pool.cc:225] Handling call
0110 02:51:34.365698 (+ 963us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.366197 (+ 499us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.385001 (+ 18804us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.385088 (+ 87us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.385155 (+ 67us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.377688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.377792 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.377796 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:34.377798 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.377865 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.377874 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.377880 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.377936 (+ 56us) tablet.cc:698] Row locks acquired
0110 02:51:34.377938 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.377961 (+ 23us) write_op.cc:270] Start()
0110 02:51:34.377982 (+ 21us) write_op.cc:276] Timestamp: P: 1736477494377916 usec, L: 0
0110 02:51:34.377985 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.378092 (+ 107us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.384178 (+ 6086us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.384289 (+ 111us) write_op.cc:301] APPLY: starting
0110 02:51:34.384299 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.384326 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.384362 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.384435 (+ 73us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.384457 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.384463 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:34.384506 (+ 43us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.384548 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.384560 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:34.384564 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.384609 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.384612 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":314,"apply.run_wall_time_us":314,"num_ops":2,"prepare.queue_time_us":10996,"prepare.run_cpu_time_us":457,"prepare.run_wall_time_us":457,"replication_time_us":6173}]]}
I20250110 02:51:34.385890 1440 master_service.cc:946] ReplaceTablet: received request to replace tablet 8435d5dd16b146c98ae86293a074445f from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:34.389151 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 188) took 2798 ms. Trace:
I20250110 02:51:34.389279 1457 rpcz_store.cc:276] 0110 02:51:31.591042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.350166 (+2759124us) service_pool.cc:225] Handling call
0110 02:51:34.372649 (+ 22483us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.373107 (+ 458us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.388939 (+ 15832us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.389028 (+ 89us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.389094 (+ 66us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.378171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.378258 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.378263 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:34.378267 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:34.378350 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.378357 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.378361 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.379845 (+ 1484us) tablet.cc:698] Row locks acquired
0110 02:51:34.379848 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:34.379874 (+ 26us) write_op.cc:270] Start()
0110 02:51:34.379897 (+ 23us) write_op.cc:276] Timestamp: P: 1736477494379828 usec, L: 0
0110 02:51:34.379900 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.380406 (+ 506us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.384228 (+ 3822us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.384653 (+ 425us) write_op.cc:301] APPLY: starting
0110 02:51:34.384660 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.384682 (+ 22us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.384705 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.384779 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.384798 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.384804 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:34.384839 (+ 35us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.384876 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.384887 (+ 11us) write_op.cc:454] Released schema lock
0110 02:51:34.384891 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.384933 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.384936 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":404,"apply.run_cpu_time_us":273,"apply.run_wall_time_us":273,"num_ops":2,"prepare.queue_time_us":4658,"prepare.run_cpu_time_us":562,"prepare.run_wall_time_us":2290,"replication_time_us":4317}]]}
I20250110 02:51:34.390498 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e804fe33c8ae489bbc4d3765c64b59bd (table default.Table_2d364ce704a5437e83b3001938e425cd [id=be22657364944595afdb47443c480b1d]) successfully deleted
I20250110 02:51:34.390416 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_18cda17085f341fa978738533448381a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.390903 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18cda17085f341fa978738533448381a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 02:51:34.391019 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6598311f59c6476ba339b8e57961cf67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6598311f59c6476ba339b8e57961cf67 already in progress: creating tablet
I20250110 02:51:34.391650 711 leader_election.cc:304] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:34.390049 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 189) took 2793 ms. Trace:
I20250110 02:51:34.391870 1450 rpcz_store.cc:276] 0110 02:51:31.597035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.363778 (+2766743us) service_pool.cc:225] Handling call
0110 02:51:34.365387 (+ 1609us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.365844 (+ 457us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.389853 (+ 24009us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.389934 (+ 81us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.389996 (+ 62us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.377094 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.377189 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.377204 (+ 15us) write_op.cc:435] Acquired schema lock
0110 02:51:34.377208 (+ 4us) tablet.cc:659] Decoding operations
0110 02:51:34.377284 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.377295 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.377300 (+ 5us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.377361 (+ 61us) tablet.cc:698] Row locks acquired
0110 02:51:34.377363 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.377394 (+ 31us) write_op.cc:270] Start()
0110 02:51:34.377420 (+ 26us) write_op.cc:276] Timestamp: P: 1736477494377344 usec, L: 0
0110 02:51:34.377423 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.377561 (+ 138us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.380491 (+ 2930us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.380576 (+ 85us) write_op.cc:301] APPLY: starting
0110 02:51:34.380588 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.380641 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.380668 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.380780 (+ 112us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.380809 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.380816 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:34.380876 (+ 60us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.380931 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.380944 (+ 13us) write_op.cc:454] Released schema lock
0110 02:51:34.380949 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.381007 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.381010 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":423,"num_ops":2,"prepare.queue_time_us":10787,"prepare.run_cpu_time_us":568,"prepare.run_wall_time_us":568,"raft.queue_time_us":31,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":219,"replication_time_us":3042}]]}
I20250110 02:51:34.392751 1450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_d62579431a1d413a9dfe71ac70e1da68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.393260 1450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d62579431a1d413a9dfe71ac70e1da68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.395190 1188 tablet_bootstrap.cc:492] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:34.395354 1188 ts_tablet_manager.cc:1397] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250110 02:51:34.396036 1188 raft_consensus.cc:357] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.396253 1188 raft_consensus.cc:383] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.396378 1188 raft_consensus.cc:738] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.396651 1188 consensus_queue.cc:260] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.397312 1188 ts_tablet_manager.cc:1428] T 7c55788eeddb47aa8a5893703c4d5e67 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:34.397645 1188 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:34.399083 1188 tablet_bootstrap.cc:654] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.402563 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df15c10ef2d4bf99af06a860c6f250f" candidate_uuid: "8899dd1aeb384e4ea4523f5fd225c888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
I20250110 02:51:34.407984 2378 raft_consensus.cc:491] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.408090 2378 raft_consensus.cc:513] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } }
I20250110 02:51:34.408581 2378 leader_election.cc:290] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.408854 915 raft_consensus.cc:3054] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.409425 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98336b684b41407da8a1949e30cd82e0" is_pre_election: true
I20250110 02:51:34.409729 772 raft_consensus.cc:2388] T b74beb77505f472fb6b9e4a30bfe6582 P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 8899dd1aeb384e4ea4523f5fd225c888 in this term.
W20250110 02:51:34.416810 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f0d4d6e02b1d4334a9993d320217f1f8 because the tablet was not found. No further retry: Not found: Tablet not found: f0d4d6e02b1d4334a9993d320217f1f8
W20250110 02:51:34.417549 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet e4726347afd649afa942d74df8c013fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4726347afd649afa942d74df8c013fe already in progress: creating tablet
I20250110 02:51:34.419486 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet fe86be26d1b544c08b231f20085d3159 (table default.Table_ed1ab72a595140a98cef7ad004d2c981 [id=621d3211bd1649a0aab67ae4c000f40d]) successfully deleted
I20250110 02:51:34.419741 853 leader_election.cc:304] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:34.421350 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
I20250110 02:51:34.422428 3294 tablet.cc:1721] T 4c0b39e2a5a4467ba44926c26d2fd1b2 P 85f9597077934f1a9814d284ed06d13d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 02:51:34.426033 2378 raft_consensus.cc:2743] T 213939cf13354ce380fb7cda5b38676f P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.428408 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet e55ee565b7f14936a2cb4cbed716c153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e55ee565b7f14936a2cb4cbed716c153 already in progress: creating tablet
I20250110 02:51:34.430254 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 8435d5dd16b146c98ae86293a074445f
I20250110 02:51:34.430752 1440 catalog_manager.cc:6596] ReplaceTablet: tablet 8435d5dd16b146c98ae86293a074445f deleted and replaced by tablet 0f3842fb666c4c81b66d708df1a5c362
I20250110 02:51:34.430879 1440 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 191) took 2771 ms. Trace:
I20250110 02:51:34.430975 1440 rpcz_store.cc:276] 0110 02:51:31.659769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.385825 (+2726056us) service_pool.cc:225] Handling call
0110 02:51:34.430827 (+ 45002us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.420992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.421111 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.421118 (+ 7us) write_op.cc:435] Acquired schema lock
0110 02:51:34.421121 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:34.421251 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.421266 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.421274 (+ 8us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.421366 (+ 92us) tablet.cc:698] Row locks acquired
0110 02:51:34.421368 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.421413 (+ 45us) write_op.cc:270] Start()
0110 02:51:34.421445 (+ 32us) write_op.cc:276] Timestamp: P: 1736477494421361 usec, L: 0
0110 02:51:34.421448 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.421656 (+ 208us) log.cc:844] Serialized 622 byte log entry
0110 02:51:34.428579 (+ 6923us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.428781 (+ 202us) write_op.cc:301] APPLY: starting
0110 02:51:34.428798 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.428850 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.428881 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.428995 (+ 114us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.429063 (+ 68us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:34.429074 (+ 11us) write_op.cc:312] APPLY: finished
0110 02:51:34.429146 (+ 72us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.429224 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.429242 (+ 18us) write_op.cc:454] Released schema lock
0110 02:51:34.429247 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.429661 (+ 414us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.429667 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":140,"apply.run_cpu_time_us":524,"apply.run_wall_time_us":524,"num_ops":2,"prepare.queue_time_us":34150,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":8666,"raft.queue_time_us":4688,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":209,"replication_time_us":7106}]]}
I20250110 02:51:34.431565 2357 tablet_replica.cc:331] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:34.431617 1440 master_service.cc:946] ReplaceTablet: received request to replace tablet e8b2297377bc4e5095826e30523eede7 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:34.431775 2357 raft_consensus.cc:2238] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.431914 2357 raft_consensus.cc:2267] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.433597 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8435d5dd16b146c98ae86293a074445f with delete_type TABLET_DATA_DELETED (replaced by tablet 0f3842fb666c4c81b66d708df1a5c362) from {username='slave'} at 127.0.0.1:43876
W20250110 02:51:34.340466 1439 rpcz_store.cc:269] 0110 02:51:33.069521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.124022 (+ 54501us) service_pool.cc:225] Handling call
0110 02:51:34.340035 (+1216013us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:33.177220 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:33.177377 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:33.177381 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:33.177383 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:33.177449 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:33.177456 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:33.177459 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:33.177631 (+ 172us) tablet.cc:698] Row locks acquired
0110 02:51:33.177633 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:33.177658 (+ 25us) write_op.cc:270] Start()
0110 02:51:33.177681 (+ 23us) write_op.cc:276] Timestamp: P: 1736477493177632 usec, L: 0
0110 02:51:33.177685 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:33.177791 (+ 106us) log.cc:844] Serialized 356 byte log entry
0110 02:51:33.277625 (+ 99834us) op_driver.cc:464] REPLICATION: finished
0110 02:51:33.282583 (+ 4958us) write_op.cc:301] APPLY: starting
0110 02:51:33.282593 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:33.282639 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:33.282660 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:33.282694 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:33.282716 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:33.282723 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:33.282771 (+ 48us) log.cc:844] Serialized 23 byte log entry
0110 02:51:33.282803 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:33.282812 (+ 9us) write_op.cc:454] Released schema lock
0110 02:51:33.282817 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 02:51:34.114848 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.114967 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.114973 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:34.114976 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:34.115098 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.115109 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.115113 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 02:51:34.115180 (+ 67us) tablet.cc:698] Row locks acquired
0110 02:51:34.115182 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.115214 (+ 32us) write_op.cc:270] Start()
0110 02:51:34.115242 (+ 28us) write_op.cc:276] Timestamp: P: 1736477494115192 usec, L: 0
0110 02:51:34.115245 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.115431 (+ 186us) log.cc:844] Serialized 356 byte log entry
0110 02:51:34.162973 (+ 47542us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.172379 (+ 9406us) write_op.cc:301] APPLY: starting
0110 02:51:34.172398 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.172445 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.172482 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.172537 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.172574 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:34.172584 (+ 10us) write_op.cc:312] APPLY: finished
0110 02:51:34.173269 (+ 685us) log.cc:844] Serialized 23 byte log entry
0110 02:51:34.173346 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.173366 (+ 20us) write_op.cc:454] Released schema lock
0110 02:51:34.173387 (+ 21us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":4926,"apply.run_cpu_time_us":278,"apply.run_wall_time_us":278,"num_ops":1,"prepare.queue_time_us":3188,"prepare.run_cpu_time_us":451,"prepare.run_wall_time_us":619,"replication_time_us":99919}],["op",{"apply.queue_time_us":9292,"apply.run_cpu_time_us":1110,"apply.run_wall_time_us":1109,"num_ops":1,"prepare.queue_time_us":224700,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":2290,"raft.queue_time_us":63,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":460,"replication_time_us":47685,"thread_start_us":239189,"threads_started":2}]]}
I20250110 02:51:34.443189 1184 tablet_bootstrap.cc:492] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.443336 2357 ts_tablet_manager.cc:1905] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.443845 1439 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_03ed6559f9ff4305b5de9549323e1aca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 02:51:34.444300 1439 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ed6559f9ff4305b5de9549323e1aca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.449705 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 192) took 2560 ms. Trace:
I20250110 02:51:34.449800 1457 rpcz_store.cc:276] 0110 02:51:31.889370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.390188 (+2500818us) service_pool.cc:225] Handling call
0110 02:51:34.391051 (+ 863us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.391446 (+ 395us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.449444 (+ 57998us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.449568 (+ 124us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.449654 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.429744 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.429833 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.429839 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:34.429841 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.438844 (+ 9003us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.438858 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.438864 (+ 6us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.438936 (+ 72us) tablet.cc:698] Row locks acquired
0110 02:51:34.438938 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.438969 (+ 31us) write_op.cc:270] Start()
0110 02:51:34.438995 (+ 26us) write_op.cc:276] Timestamp: P: 1736477494438917 usec, L: 0
0110 02:51:34.438999 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.439135 (+ 136us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.448782 (+ 9647us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.448876 (+ 94us) write_op.cc:301] APPLY: starting
0110 02:51:34.448891 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.448933 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.448962 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.449078 (+ 116us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.449115 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.449122 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:34.449191 (+ 69us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.449266 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.449283 (+ 17us) write_op.cc:454] Released schema lock
0110 02:51:34.449289 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.449358 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.449362 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":473,"num_ops":2,"prepare.queue_time_us":37902,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":10559,"raft.queue_time_us":268,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"replication_time_us":9758,"spinlock_wait_cycles":56576}]]}
I20250110 02:51:34.451910 3224 raft_consensus.cc:2798] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 02:51:34.451983 3224 raft_consensus.cc:491] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 02:51:34.452037 3224 raft_consensus.cc:3054] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.455282 1450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 193) took 2535 ms. Trace:
I20250110 02:51:34.455363 1450 rpcz_store.cc:276] 0110 02:51:31.919661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.392564 (+2472903us) service_pool.cc:225] Handling call
0110 02:51:34.393363 (+ 799us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.393813 (+ 450us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.455052 (+ 61239us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.455147 (+ 95us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.455229 (+ 82us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.440341 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.440428 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.440434 (+ 6us) write_op.cc:435] Acquired schema lock
0110 02:51:34.440437 (+ 3us) tablet.cc:659] Decoding operations
0110 02:51:34.440509 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.440517 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.440521 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.440596 (+ 75us) tablet.cc:698] Row locks acquired
0110 02:51:34.440598 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.440624 (+ 26us) write_op.cc:270] Start()
0110 02:51:34.440649 (+ 25us) write_op.cc:276] Timestamp: P: 1736477494440572 usec, L: 0
0110 02:51:34.440652 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.440797 (+ 145us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.452598 (+ 11801us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.452995 (+ 397us) write_op.cc:301] APPLY: starting
0110 02:51:34.453007 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.453037 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.453057 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.453144 (+ 87us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.453171 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.453177 (+ 6us) write_op.cc:312] APPLY: finished
0110 02:51:34.453241 (+ 64us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.453294 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.453306 (+ 12us) write_op.cc:454] Released schema lock
0110 02:51:34.453311 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.453369 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.453372 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":349,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":371,"num_ops":2,"prepare.queue_time_us":46091,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":518,"replication_time_us":11926}]]}
I20250110 02:51:34.455960 1450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3352f8378b9846a3854086293f2d4e15 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:34.456311 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet e8b2297377bc4e5095826e30523eede7
I20250110 02:51:34.456796 1440 catalog_manager.cc:6596] ReplaceTablet: tablet e8b2297377bc4e5095826e30523eede7 deleted and replaced by tablet 9dceba47ef44407bbf53e70504c43e5c
I20250110 02:51:34.456883 1440 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 194) took 2520 ms. Trace:
I20250110 02:51:34.456964 1440 rpcz_store.cc:276] 0110 02:51:31.936717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.431543 (+2494826us) service_pool.cc:225] Handling call
0110 02:51:34.456858 (+ 25315us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.440896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.440983 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.440987 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:34.440989 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.441072 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.441080 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.441084 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.441164 (+ 80us) tablet.cc:698] Row locks acquired
0110 02:51:34.441166 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.441188 (+ 22us) write_op.cc:270] Start()
0110 02:51:34.441218 (+ 30us) write_op.cc:276] Timestamp: P: 1736477494441138 usec, L: 0
0110 02:51:34.441222 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.441353 (+ 131us) log.cc:844] Serialized 656 byte log entry
0110 02:51:34.452650 (+ 11297us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.453443 (+ 793us) write_op.cc:301] APPLY: starting
0110 02:51:34.453469 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.453498 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.453525 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.453601 (+ 76us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.453644 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:34.453651 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:34.453698 (+ 47us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.453749 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.453763 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:34.453768 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.453821 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.453824 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1934080,"child_traces":[["op",{"apply.queue_time_us":764,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":371,"num_ops":2,"prepare.queue_time_us":7387,"prepare.run_cpu_time_us":509,"prepare.run_wall_time_us":508,"replication_time_us":11415}]]}
I20250110 02:51:34.457454 1440 master_service.cc:946] ReplaceTablet: received request to replace tablet 7efa857721c34003ad91912ef2d7b381 from {username='slave'} at 127.0.0.1:51672
I20250110 02:51:34.458845 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b2297377bc4e5095826e30523eede7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9dceba47ef44407bbf53e70504c43e5c) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.460635 1457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51672:
name: "default.Table_9cb7bb66e3be4cae9b02983427b8c065"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 02:51:34.461159 3315 raft_consensus.cc:491] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 02:51:34.461213 1457 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cb7bb66e3be4cae9b02983427b8c065 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 02:51:34.461292 3315 raft_consensus.cc:513] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.461879 3315 leader_election.cc:290] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.462175 1439 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 195) took 2511 ms. Trace:
I20250110 02:51:34.462270 3315 raft_consensus.cc:491] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.462255 1439 rpcz_store.cc:276] 0110 02:51:31.950556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.443603 (+2493047us) service_pool.cc:225] Handling call
0110 02:51:34.444390 (+ 787us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.444780 (+ 390us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.461911 (+ 17131us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.462006 (+ 95us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.462105 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.455504 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.455590 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.455595 (+ 5us) write_op.cc:435] Acquired schema lock
0110 02:51:34.455597 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.455668 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.455677 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.455681 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.455739 (+ 58us) tablet.cc:698] Row locks acquired
0110 02:51:34.455741 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.455766 (+ 25us) write_op.cc:270] Start()
0110 02:51:34.455787 (+ 21us) write_op.cc:276] Timestamp: P: 1736477494455713 usec, L: 0
0110 02:51:34.455791 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.455921 (+ 130us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.460971 (+ 5050us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.461414 (+ 443us) write_op.cc:301] APPLY: starting
0110 02:51:34.461426 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.461460 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.461481 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.461583 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.461609 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.461617 (+ 8us) write_op.cc:312] APPLY: finished
0110 02:51:34.461675 (+ 58us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.461731 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.461747 (+ 16us) write_op.cc:454] Released schema lock
0110 02:51:34.461753 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.461812 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.461815 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":392,"apply.run_cpu_time_us":395,"apply.run_wall_time_us":395,"num_ops":2,"prepare.queue_time_us":10323,"prepare.run_cpu_time_us":521,"prepare.run_wall_time_us":1610,"raft.queue_time_us":33,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":180,"replication_time_us":5156}]]}
I20250110 02:51:34.462795 1439 master_service.cc:946] ReplaceTablet: received request to replace tablet eadb829d657346eabcb7d2aa34b33fe2 from {username='slave'} at 127.0.0.1:51672
W20250110 02:51:34.463514 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:35771, user_credentials={real_user=slave}} blocked reactor thread for 71893.3us
I20250110 02:51:34.463714 3311 raft_consensus.cc:491] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.463829 3311 raft_consensus.cc:513] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.464282 3311 leader_election.cc:290] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.464430 3311 raft_consensus.cc:491] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.464514 3311 raft_consensus.cc:513] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.464916 3311 leader_election.cc:290] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.443368 1184 ts_tablet_manager.cc:1397] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.372s user 0.020s sys 0.038s
I20250110 02:51:34.466770 3224 raft_consensus.cc:513] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.466791 1184 raft_consensus.cc:357] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.466979 1184 raft_consensus.cc:383] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.467024 1184 raft_consensus.cc:738] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.467248 3224 leader_election.cc:290] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.467185 1184 consensus_queue.cc:260] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.467649 1184 ts_tablet_manager.cc:1428] T 3d4f88a7002f413fbd63491fe8e1fe6d P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 02:51:34.467898 1184 tablet_bootstrap.cc:492] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.469522 1188 tablet_bootstrap.cc:492] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d: No bootstrap required, opened a new log
I20250110 02:51:34.469650 1188 ts_tablet_manager.cc:1397] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.002s
W20250110 02:51:34.471693 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4de67338a26f4febb5dd7611a72b4723 because the tablet was not found. No further retry: Not found: Tablet not found: 4de67338a26f4febb5dd7611a72b4723
I20250110 02:51:34.471750 1440 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet 7efa857721c34003ad91912ef2d7b381
W20250110 02:51:34.472090 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet 4f015e99f53f48599ac2d7a29220bc39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f015e99f53f48599ac2d7a29220bc39 already in progress: creating tablet
I20250110 02:51:34.472189 1440 catalog_manager.cc:6596] ReplaceTablet: tablet 7efa857721c34003ad91912ef2d7b381 deleted and replaced by tablet ae118460f3e8429ea62fb5a7c5e36761
I20250110 02:51:34.472314 1440 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 198) took 2389 ms. Trace:
W20250110 02:51:34.472589 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet bb595013e809480a814cb9c7aed856e6 because the tablet was not found. No further retry: Not found: Tablet not found: bb595013e809480a814cb9c7aed856e6
I20250110 02:51:34.472396 1440 rpcz_store.cc:276] 0110 02:51:32.082626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.457405 (+2374779us) service_pool.cc:225] Handling call
0110 02:51:34.472264 (+ 14859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.464198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.464296 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.464300 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:34.464302 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.464427 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.464438 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.464442 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.464539 (+ 97us) tablet.cc:698] Row locks acquired
0110 02:51:34.464541 (+ 2us) write_op.cc:260] PREPARE: finished
0110 02:51:34.464569 (+ 28us) write_op.cc:270] Start()
0110 02:51:34.464593 (+ 24us) write_op.cc:276] Timestamp: P: 1736477494464515 usec, L: 0
0110 02:51:34.464596 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.464755 (+ 159us) log.cc:844] Serialized 656 byte log entry
0110 02:51:34.470984 (+ 6229us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.471074 (+ 90us) write_op.cc:301] APPLY: starting
0110 02:51:34.471087 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.471138 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.471165 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.471267 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.471316 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:34.471323 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:34.471380 (+ 57us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.471435 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.471450 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:34.471456 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"apply.run_cpu_time_us":442,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":5914,"raft.queue_time_us":33,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":197,"replication_time_us":6362}]]}
I20250110 02:51:34.473271 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efa857721c34003ad91912ef2d7b381 with delete_type TABLET_DATA_DELETED (replaced by tablet ae118460f3e8429ea62fb5a7c5e36761) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.473325 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8435d5dd16b146c98ae86293a074445f with delete_type TABLET_DATA_DELETED (replaced by tablet 0f3842fb666c4c81b66d708df1a5c362) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.473506 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b2297377bc4e5095826e30523eede7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9dceba47ef44407bbf53e70504c43e5c) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.473635 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efa857721c34003ad91912ef2d7b381 with delete_type TABLET_DATA_DELETED (replaced by tablet ae118460f3e8429ea62fb5a7c5e36761) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.474299 1440 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_2d364ce704a5437e83b3001938e425cd" } new_table_name: "default.Table_4207c9caf64e476db10216a55743ee41" modify_external_catalogs: true
I20250110 02:51:34.475059 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa78a9ef5eb443da0c8983a7190440b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.475708 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb805b3e537466ab7fe0185421ad5aa" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.476751 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da1ee1add01482cbf651bd2b4753dc1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.477310 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2b5a305db34fedbbbf0d666e6c28cb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d"
W20250110 02:51:34.426012 994 leader_election.cc:343] T b48fe2d061da4e729c8704af4fc83f7a P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.477433 2224 rpcz_store.cc:276] 0110 02:51:32.999099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:32.999395 (+ 296us) service_pool.cc:225] Handling call
0110 02:51:34.360562 (+1361167us) ts_tablet_manager.cc:1145] Deleting tablet 6e0b2ec1d5814144bc58261e6c6e8d35
0110 02:51:34.360572 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.360629 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360996}
W20250110 02:51:34.477748 994 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33221, user_credentials={real_user=slave}} blocked reactor thread for 51744.3us
I20250110 02:51:34.435179 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca45a5c0b3cb4b53862e51b11c4d3a00" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.478060 1061 raft_consensus.cc:2388] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85f9597077934f1a9814d284ed06d13d in current term 1: Already voted for candidate 98336b684b41407da8a1949e30cd82e0 in this term.
I20250110 02:51:34.435874 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74beb77505f472fb6b9e4a30bfe6582" candidate_uuid: "85f9597077934f1a9814d284ed06d13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.455338 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8435d5dd16b146c98ae86293a074445f with delete_type TABLET_DATA_DELETED (replaced by tablet 0f3842fb666c4c81b66d708df1a5c362) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:34.466101 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa78a9ef5eb443da0c8983a7190440b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.468443 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet e8b2297377bc4e5095826e30523eede7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9dceba47ef44407bbf53e70504c43e5c) from {username='slave'} at 127.0.0.1:37598
W20250110 02:51:34.478726 994 leader_election.cc:343] T 1df15c10ef2d4bf99af06a860c6f250f P 8899dd1aeb384e4ea4523f5fd225c888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:34.478992 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6e0b2ec1d5814144bc58261e6c6e8d35 because the tablet was not found. No further retry: Not found: Tablet not found: 6e0b2ec1d5814144bc58261e6c6e8d35
I20250110 02:51:34.429787 1181 tablet_bootstrap.cc:492] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.479383 1181 ts_tablet_manager.cc:1397] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.276s user 0.008s sys 0.013s
I20250110 02:51:34.480497 1181 raft_consensus.cc:357] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.480671 1181 raft_consensus.cc:383] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.480718 1181 raft_consensus.cc:738] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.480926 1181 consensus_queue.cc:260] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.481475 1181 ts_tablet_manager.cc:1428] T e9abd7180bf64bb5991be5f67ea42fbf P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:34.482012 1181 tablet_bootstrap.cc:492] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.482422 1457 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51672 (request call id 196) took 2527 ms. Trace:
I20250110 02:51:34.482510 1457 rpcz_store.cc:276] 0110 02:51:31.954978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.450307 (+2495329us) service_pool.cc:225] Handling call
0110 02:51:34.461292 (+ 10985us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 02:51:34.461690 (+ 398us) catalog_manager.cc:2122] Created new table and tablet info
0110 02:51:34.482157 (+ 20467us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 02:51:34.482279 (+ 122us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 02:51:34.482356 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.476979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.477065 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.477069 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:34.477071 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.477141 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.477150 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.477154 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.477237 (+ 83us) tablet.cc:698] Row locks acquired
0110 02:51:34.477240 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:34.477266 (+ 26us) write_op.cc:270] Start()
0110 02:51:34.477288 (+ 22us) write_op.cc:276] Timestamp: P: 1736477494477212 usec, L: 0
0110 02:51:34.477291 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.477427 (+ 136us) log.cc:844] Serialized 433 byte log entry
0110 02:51:34.480127 (+ 2700us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.480284 (+ 157us) write_op.cc:301] APPLY: starting
0110 02:51:34.480295 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.480326 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.480346 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.480423 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.480448 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 02:51:34.480453 (+ 5us) write_op.cc:312] APPLY: finished
0110 02:51:34.480509 (+ 56us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.480567 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.480581 (+ 14us) write_op.cc:454] Released schema lock
0110 02:51:34.480585 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":352,"apply.run_wall_time_us":352,"num_ops":2,"prepare.queue_time_us":14830,"raft.queue_time_us":30,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"replication_time_us":2812}]]}
I20250110 02:51:34.483527 1457 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2" } modify_external_catalogs: true
I20250110 02:51:34.483675 1457 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_3ec8e056dca34c1b91651cbc6ca2b9c2" } modify_external_catalogs: true
I20250110 02:51:34.483538 912 raft_consensus.cc:3054] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.477771 2224 tablet_replica.cc:331] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
W20250110 02:51:34.489967 709 leader_election.cc:343] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.491673 1181 tablet_bootstrap.cc:654] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.491837 2224 raft_consensus.cc:2238] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:34.492157 2224 pending_rounds.cc:62] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Trying to abort 1 pending ops.
I20250110 02:51:34.492262 2224 pending_rounds.cc:69] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112611801814040576 op_type: NO_OP noop_request { }
I20250110 02:51:34.492409 2224 raft_consensus.cc:2883] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 02:51:34.492547 2224 raft_consensus.cc:2267] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.493311 2224 ts_tablet_manager.cc:1905] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.495762 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efa857721c34003ad91912ef2d7b381 with delete_type TABLET_DATA_DELETED (replaced by tablet ae118460f3e8429ea62fb5a7c5e36761) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:34.495848 853 leader_election.cc:304] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:34.496094 2378 raft_consensus.cc:2743] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.496277 853 leader_election.cc:304] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
I20250110 02:51:34.496475 2378 raft_consensus.cc:3054] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.499125 2378 raft_consensus.cc:2743] T b74beb77505f472fb6b9e4a30bfe6582 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.499203 3315 raft_consensus.cc:513] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
W20250110 02:51:34.495616 709 leader_election.cc:343] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.500941 1188 raft_consensus.cc:357] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.501713 1188 raft_consensus.cc:383] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.502174 1188 raft_consensus.cc:738] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f9597077934f1a9814d284ed06d13d, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.503247 1188 consensus_queue.cc:260] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.500030 3315 leader_election.cc:290] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.504338 1188 ts_tablet_manager.cc:1428] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20250110 02:51:34.504642 1188 tablet_bootstrap.cc:492] T a7d4f03a8c66446ea7c6d178cc7ef80a P 85f9597077934f1a9814d284ed06d13d: Bootstrap starting.
I20250110 02:51:34.505187 1184 tablet_bootstrap.cc:654] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.506283 1188 tablet_bootstrap.cc:654] T a7d4f03a8c66446ea7c6d178cc7ef80a P 85f9597077934f1a9814d284ed06d13d: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.507520 2352 ts_tablet_manager.cc:1918] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:34.507612 2352 log.cc:1198] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/dba0cb7970f1453dac256b255ed7cac0
I20250110 02:51:34.507889 2352 ts_tablet_manager.cc:1939] T dba0cb7970f1453dac256b255ed7cac0 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
W20250110 02:51:34.500628 709 leader_election.cc:343] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
W20250110 02:51:34.504112 711 leader_election.cc:343] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.508481 711 leader_election.cc:304] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:34.508666 709 leader_election.cc:343] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.509485 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet dba0cb7970f1453dac256b255ed7cac0 (table default.Table_161f237741c449f1b10e9b240cca7ef2 [id=9fab86bd20e2470989415eaf67f4af50]) successfully deleted
I20250110 02:51:34.509989 2352 tablet_replica.cc:331] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:34.510160 2352 raft_consensus.cc:2238] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 02:51:34.510144 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet cdb726d2c9a04963b92a3fb625cf84f3 because the tablet was not found. No further retry: Not found: Tablet not found: cdb726d2c9a04963b92a3fb625cf84f3
I20250110 02:51:34.510303 2352 raft_consensus.cc:2267] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 02:51:34.510365 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet fe86be26d1b544c08b231f20085d3159 because the tablet was not found. No further retry: Not found: Tablet not found: fe86be26d1b544c08b231f20085d3159
W20250110 02:51:34.510561 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 3ca895eca14748c19fdb7d363499b544 because the tablet was not found. No further retry: Not found: Tablet not found: 3ca895eca14748c19fdb7d363499b544
W20250110 02:51:34.510730 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 80186dd7d1a240068584b64ce9e9b03f because the tablet was not found. No further retry: Not found: Tablet not found: 80186dd7d1a240068584b64ce9e9b03f
I20250110 02:51:34.510774 2352 ts_tablet_manager.cc:1905] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:34.510893 1429 catalog_manager.cc:4839] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet d51adff7f50c4aec94e1892a50ec36c4 because the tablet was not found. No further retry: Not found: Tablet not found: d51adff7f50c4aec94e1892a50ec36c4
W20250110 02:51:34.511080 1429 catalog_manager.cc:4852] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): delete failed for tablet 8435d5dd16b146c98ae86293a074445f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8435d5dd16b146c98ae86293a074445f already in progress: creating tablet
I20250110 02:51:34.511557 3224 raft_consensus.cc:2743] T 8600a637306741dd8e1e7f4964ff4bc8 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.508877 711 leader_election.cc:343] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.511735 711 leader_election.cc:304] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:34.511896 711 raft_consensus.cc:2681] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 02:51:34.512264 711 leader_election.cc:343] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.512353 711 leader_election.cc:304] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.512285 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b255096800d4cd58324e2c06ff1a44c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.513238 3319 raft_consensus.cc:2743] T 4aa78a9ef5eb443da0c8983a7190440b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.513535 709 leader_election.cc:343] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.514223 2289 consensus_queue.cc:1035] T 24cf81039d5e4a47b426ea5d2f21acc2 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:34.514410 3276 consensus_queue.cc:1035] T b74beb77505f472fb6b9e4a30bfe6582 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:34.515522 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb805b3e537466ab7fe0185421ad5aa" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.516016 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da1ee1add01482cbf651bd2b4753dc1" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.516405 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2b5a305db34fedbbbf0d666e6c28cb" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888"
I20250110 02:51:34.516525 1056 raft_consensus.cc:3054] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.519109 1056 raft_consensus.cc:2463] T aa2b5a305db34fedbbbf0d666e6c28cb P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98336b684b41407da8a1949e30cd82e0 in term 1.
I20250110 02:51:34.520278 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b255096800d4cd58324e2c06ff1a44c" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.522306 1439 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5d8d40556dc348e8aa8255d116251a10: Sending DeleteTablet for 3 replicas of tablet eadb829d657346eabcb7d2aa34b33fe2
I20250110 02:51:34.528858 2378 raft_consensus.cc:491] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.529002 2378 raft_consensus.cc:513] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.529636 2378 leader_election.cc:290] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139)
W20250110 02:51:34.530681 853 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Not found: Tablet not found: eed058931194480ca1a27130bfa8ee32
W20250110 02:51:34.531025 853 leader_election.cc:343] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Not found: Tablet not found: eed058931194480ca1a27130bfa8ee32
I20250110 02:51:34.531116 853 leader_election.cc:304] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f9597077934f1a9814d284ed06d13d; no voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0
W20250110 02:51:34.531332 711 leader_election.cc:343] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.531463 711 leader_election.cc:304] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.531481 2378 raft_consensus.cc:2743] T eed058931194480ca1a27130bfa8ee32 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.531688 3224 raft_consensus.cc:2743] T 3fb805b3e537466ab7fe0185421ad5aa P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 02:51:34.531903 711 leader_election.cc:343] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.532016 711 leader_election.cc:304] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.532223 3315 consensus_queue.cc:1035] T ca45a5c0b3cb4b53862e51b11c4d3a00 P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:34.532323 711 leader_election.cc:304] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8899dd1aeb384e4ea4523f5fd225c888, 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d
I20250110 02:51:34.533432 908 raft_consensus.cc:3054] T 36c0aaf2050f481daa8cdac4761fe705 P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.535504 3319 raft_consensus.cc:2743] T 2da1ee1add01482cbf651bd2b4753dc1 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.535522 3315 raft_consensus.cc:2798] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 02:51:34.535666 3315 raft_consensus.cc:695] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Running, Role: LEADER
W20250110 02:51:34.535964 711 leader_election.cc:343] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.535900 3315 consensus_queue.cc:237] T aa2b5a305db34fedbbbf0d666e6c28cb P 98336b684b41407da8a1949e30cd82e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.536048 711 leader_election.cc:304] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.536240 3319 raft_consensus.cc:2743] T 8b255096800d4cd58324e2c06ff1a44c P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.537886 2224 ts_tablet_manager.cc:1918] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.540442 1184 tablet_bootstrap.cc:492] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.540659 1184 ts_tablet_manager.cc:1397] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250110 02:51:34.541672 1184 raft_consensus.cc:357] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.542038 1184 raft_consensus.cc:383] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.542193 1184 raft_consensus.cc:738] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.543408 1184 consensus_queue.cc:260] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.544085 881 tablet_service.cc:1514] Processing DeleteTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 30f4126d4b184f41899ce941eabd1195) from {username='slave'} at 127.0.0.1:43876
I20250110 02:51:34.546007 1184 ts_tablet_manager.cc:1428] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250110 02:51:34.546417 3152 consensus_queue.cc:1035] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 02:51:34.546540 2224 log.cc:1198] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/e015a41a03f84c91a45e46e4b7b5e7a9
I20250110 02:51:34.547063 2224 ts_tablet_manager.cc:1939] T e015a41a03f84c91a45e46e4b7b5e7a9 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.547396 1184 tablet_bootstrap.cc:492] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.549437 1439 catalog_manager.cc:6596] ReplaceTablet: tablet eadb829d657346eabcb7d2aa34b33fe2 deleted and replaced by tablet 30f4126d4b184f41899ce941eabd1195
I20250110 02:51:34.549577 1439 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51672 (request call id 199) took 2345 ms. Trace:
I20250110 02:51:34.547927 742 tablet_service.cc:1514] Processing DeleteTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 30f4126d4b184f41899ce941eabd1195) from {username='slave'} at 127.0.0.1:50178
I20250110 02:51:34.547168 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet eadb829d657346eabcb7d2aa34b33fe2 with delete_type TABLET_DATA_DELETED (replaced by tablet 30f4126d4b184f41899ce941eabd1195) from {username='slave'} at 127.0.0.1:37598
I20250110 02:51:34.551362 912 raft_consensus.cc:3054] T e2ab2376576448deb87f607b3e5a4f1c P 85f9597077934f1a9814d284ed06d13d [term 0 FOLLOWER]: Advancing to term 1
I20250110 02:51:34.552246 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 428) took 1553 ms. Trace:
I20250110 02:51:34.556231 3319 raft_consensus.cc:491] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.556406 3319 raft_consensus.cc:513] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.555814 2224 rpcz_store.cc:276] 0110 02:51:32.999004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.005167 (+ 6163us) service_pool.cc:225] Handling call
0110 02:51:34.477727 (+1472560us) ts_tablet_manager.cc:1145] Deleting tablet e015a41a03f84c91a45e46e4b7b5e7a9
0110 02:51:34.477736 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.519520 (+ 41784us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.537793 (+ 18273us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.552224 (+ 14431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472360}
I20250110 02:51:34.557037 3319 leader_election.cc:290] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
I20250110 02:51:34.557384 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb775de0154409a0d92990a51d1ee0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.558230 709 leader_election.cc:343] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.557574 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fb775de0154409a0d92990a51d1ee0" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
I20250110 02:51:34.560967 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 430) took 1561 ms. Trace:
I20250110 02:51:34.561054 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet e015a41a03f84c91a45e46e4b7b5e7a9 (table default.Table_a1daaf2964134078b6e9ac86b7672a37 [id=f922721df2054dd6aa157270e96733b6]) successfully deleted
I20250110 02:51:34.517158 2357 ts_tablet_manager.cc:1918] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:34.561674 2357 log.cc:1198] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/bc8444c1381a45d6b2fee30da971bd47
W20250110 02:51:34.561803 711 leader_election.cc:343] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.561980 1184 tablet_bootstrap.cc:654] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.562215 2224 rpcz_store.cc:276] 0110 02:51:32.999169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.005398 (+ 6229us) service_pool.cc:225] Handling call
0110 02:51:34.560887 (+1555489us) ts_tablet_manager.cc:1145] Deleting tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa
0110 02:51:34.560895 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.560954 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555347}
I20250110 02:51:34.562840 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 431) took 1563 ms. Trace:
W20250110 02:51:34.562942 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet c2b2aa3a302a47d3a3cb4dfc2e330cfa because the tablet was not found. No further retry: Not found: Tablet not found: c2b2aa3a302a47d3a3cb4dfc2e330cfa
I20250110 02:51:34.549693 1439 rpcz_store.cc:276] 0110 02:51:32.203728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:34.462734 (+2259006us) service_pool.cc:225] Handling call
0110 02:51:34.549546 (+ 86812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 02:51:34.485351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 02:51:34.485432 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0110 02:51:34.485436 (+ 4us) write_op.cc:435] Acquired schema lock
0110 02:51:34.485438 (+ 2us) tablet.cc:659] Decoding operations
0110 02:51:34.485521 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 02:51:34.485531 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 02:51:34.485535 (+ 4us) tablet.cc:682] Acquiring locks for 2 operations
0110 02:51:34.485590 (+ 55us) tablet.cc:698] Row locks acquired
0110 02:51:34.485593 (+ 3us) write_op.cc:260] PREPARE: finished
0110 02:51:34.485614 (+ 21us) write_op.cc:270] Start()
0110 02:51:34.485647 (+ 33us) write_op.cc:276] Timestamp: P: 1736477494485560 usec, L: 0
0110 02:51:34.485650 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 02:51:34.485807 (+ 157us) log.cc:844] Serialized 656 byte log entry
0110 02:51:34.488792 (+ 2985us) op_driver.cc:464] REPLICATION: finished
0110 02:51:34.488961 (+ 169us) write_op.cc:301] APPLY: starting
0110 02:51:34.488973 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 02:51:34.489005 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0110 02:51:34.489039 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 02:51:34.489117 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 02:51:34.489159 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 02:51:34.489166 (+ 7us) write_op.cc:312] APPLY: finished
0110 02:51:34.489231 (+ 65us) log.cc:844] Serialized 29 byte log entry
0110 02:51:34.489284 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.489299 (+ 15us) write_op.cc:454] Released schema lock
0110 02:51:34.489304 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0110 02:51:34.491322 (+ 2018us) write_op.cc:489] Releasing partition, row and schema locks
0110 02:51:34.491327 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":131,"apply.run_cpu_time_us":393,"apply.run_wall_time_us":393,"num_ops":2,"prepare.queue_time_us":21770,"prepare.run_cpu_time_us":555,"prepare.run_wall_time_us":5964,"raft.queue_time_us":670,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":169,"replication_time_us":3118}]]}
I20250110 02:51:34.566030 1439 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_df7558651391471ba93ced530307d670" } modify_external_catalogs: true
I20250110 02:51:34.567119 2224 rpcz_store.cc:276] 0110 02:51:32.999257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.005567 (+ 6310us) service_pool.cc:225] Handling call
0110 02:51:34.562746 (+1557179us) ts_tablet_manager.cc:1145] Deleting tablet 6555b0a4491046bf8f5cd4ee406c3955
0110 02:51:34.562767 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.562824 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557035}
I20250110 02:51:34.567552 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 434) took 1559 ms. Trace:
W20250110 02:51:34.563556 994 consensus_peers.cc:487] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:34.563107 1181 tablet_bootstrap.cc:492] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.568830 2224 rpcz_store.cc:276] 0110 02:51:33.008129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.008182 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:34.567477 (+1559295us) ts_tablet_manager.cc:1145] Deleting tablet 19e4af23720048fe98b3231ba402d45c
0110 02:51:34.567486 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.567539 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559092}
W20250110 02:51:34.567571 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 6555b0a4491046bf8f5cd4ee406c3955 because the tablet was not found. No further retry: Not found: Tablet not found: 6555b0a4491046bf8f5cd4ee406c3955
I20250110 02:51:34.569197 1181 ts_tablet_manager.cc:1397] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.007s
I20250110 02:51:34.569263 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 436) took 1560 ms. Trace:
I20250110 02:51:34.569350 2224 rpcz_store.cc:276] 0110 02:51:33.008338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.008778 (+ 440us) service_pool.cc:225] Handling call
0110 02:51:34.569162 (+1560384us) ts_tablet_manager.cc:1145] Deleting tablet b382d634db9642df8e00c5ccaa5787a9
0110 02:51:34.569187 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.569249 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560220}
W20250110 02:51:34.569496 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 19e4af23720048fe98b3231ba402d45c because the tablet was not found. No further retry: Not found: Tablet not found: 19e4af23720048fe98b3231ba402d45c
I20250110 02:51:34.569651 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 441) took 1560 ms. Trace:
W20250110 02:51:34.569739 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b382d634db9642df8e00c5ccaa5787a9 because the tablet was not found. No further retry: Not found: Tablet not found: b382d634db9642df8e00c5ccaa5787a9
I20250110 02:51:34.569730 2224 rpcz_store.cc:276] 0110 02:51:33.008765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.008980 (+ 215us) service_pool.cc:225] Handling call
0110 02:51:34.569559 (+1560579us) ts_tablet_manager.cc:1145] Deleting tablet 8d1f2acd39be40e4b71cc0dcf68e7567
0110 02:51:34.569567 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.569639 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560411}
I20250110 02:51:34.570008 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 442) took 1561 ms. Trace:
W20250110 02:51:34.570039 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8d1f2acd39be40e4b71cc0dcf68e7567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1f2acd39be40e4b71cc0dcf68e7567 already in progress: creating tablet
I20250110 02:51:34.570079 2224 rpcz_store.cc:276] 0110 02:51:33.008882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.009173 (+ 291us) service_pool.cc:225] Handling call
0110 02:51:34.569941 (+1560768us) ts_tablet_manager.cc:1145] Deleting tablet 302af84fd8374ffa96a7503b54849536
0110 02:51:34.569949 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.569996 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560591}
I20250110 02:51:34.570343 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 438) took 1561 ms. Trace:
W20250110 02:51:34.570451 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 302af84fd8374ffa96a7503b54849536 because the tablet was not found. No further retry: Not found: Tablet not found: 302af84fd8374ffa96a7503b54849536
I20250110 02:51:34.570474 1181 raft_consensus.cc:357] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.570672 1181 raft_consensus.cc:383] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.570736 1181 raft_consensus.cc:738] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.570974 1181 consensus_queue.cc:260] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.569332 1439 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51672:
table { table_name: "default.Table_df7558651391471ba93ced530307d670" } modify_external_catalogs: true
I20250110 02:51:34.571528 1181 ts_tablet_manager.cc:1428] T b5134444da58427aa97e166e44a3a648 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 02:51:34.571763 1181 tablet_bootstrap.cc:492] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.574546 822 heartbeater.cc:502] Master 127.0.104.60:39025 was elected leader, sending a full tablet report...
I20250110 02:51:34.576012 1181 tablet_bootstrap.cc:654] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.570425 2224 rpcz_store.cc:276] 0110 02:51:33.008486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.009146 (+ 660us) service_pool.cc:225] Handling call
0110 02:51:34.570277 (+1561131us) ts_tablet_manager.cc:1145] Deleting tablet 4d970e2fe72b4727968ca605b3e772cc
0110 02:51:34.570286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.570329 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560915}
I20250110 02:51:34.577014 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 440) took 1568 ms. Trace:
I20250110 02:51:34.577086 2224 rpcz_store.cc:276] 0110 02:51:33.008682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.009923 (+ 1241us) service_pool.cc:225] Handling call
0110 02:51:34.576931 (+1567008us) ts_tablet_manager.cc:1145] Deleting tablet 1edb4e06e40c43438882e1eb622d788e
0110 02:51:34.576939 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.577000 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566804}
I20250110 02:51:34.577381 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 437) took 1568 ms. Trace:
I20250110 02:51:34.577467 2224 rpcz_store.cc:276] 0110 02:51:33.008418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.010214 (+ 1796us) service_pool.cc:225] Handling call
0110 02:51:34.577311 (+1567097us) ts_tablet_manager.cc:1145] Deleting tablet 9289de0630804f9aa60fdaf5f3f15f68
0110 02:51:34.577319 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.577366 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566990}
W20250110 02:51:34.577069 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 4d970e2fe72b4727968ca605b3e772cc because the tablet was not found. No further retry: Not found: Tablet not found: 4d970e2fe72b4727968ca605b3e772cc
I20250110 02:51:34.577804 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 444) took 1561 ms. Trace:
W20250110 02:51:34.577929 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 1edb4e06e40c43438882e1eb622d788e because the tablet was not found. No further retry: Not found: Tablet not found: 1edb4e06e40c43438882e1eb622d788e
W20250110 02:51:34.578125 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 9289de0630804f9aa60fdaf5f3f15f68 because the tablet was not found. No further retry: Not found: Tablet not found: 9289de0630804f9aa60fdaf5f3f15f68
I20250110 02:51:34.562168 2357 ts_tablet_manager.cc:1939] T bc8444c1381a45d6b2fee30da971bd47 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:34.582890 2357 tablet_replica.cc:331] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:34.583083 2357 raft_consensus.cc:2238] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.583264 2357 raft_consensus.cc:2267] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.583755 2357 ts_tablet_manager.cc:1905] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.561936 711 leader_election.cc:304] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.577862 2224 rpcz_store.cc:276] 0110 02:51:33.016145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.016185 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:34.577710 (+1561525us) ts_tablet_manager.cc:1145] Deleting tablet fe860d60b3194dc5a2e7a94d39d4b8dd
0110 02:51:34.577721 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.577793 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561393}
I20250110 02:51:34.585194 3319 raft_consensus.cc:491] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.585325 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 445) took 1568 ms. Trace:
I20250110 02:51:34.585362 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet bc8444c1381a45d6b2fee30da971bd47 (table default.Table_e4c35b6a442b4b95a1341303de60226f [id=13200b366f444ad596f4d42b5592e253]) successfully deleted
I20250110 02:51:34.585347 3319 raft_consensus.cc:513] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.585399 2224 rpcz_store.cc:276] 0110 02:51:33.017298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.017343 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:34.585254 (+1567911us) ts_tablet_manager.cc:1145] Deleting tablet 3510e86314144a2ca012aacf3d78eacc
0110 02:51:34.585263 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.585311 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567762}
I20250110 02:51:34.585629 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 446) took 1566 ms. Trace:
I20250110 02:51:34.585706 2224 rpcz_store.cc:276] 0110 02:51:33.019216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.019265 (+ 49us) service_pool.cc:225] Handling call
0110 02:51:34.585567 (+1566302us) ts_tablet_manager.cc:1145] Deleting tablet 323ce9f6a3bb4e4a8d80ca0b047500a5
0110 02:51:34.585574 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.585617 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566156}
I20250110 02:51:34.585928 2224 tablet_replica.cc:331] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.585994 3319 leader_election.cc:290] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771), 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221)
I20250110 02:51:34.586103 2224 raft_consensus.cc:2238] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:34.586159 3319 raft_consensus.cc:2743] T b9fb775de0154409a0d92990a51d1ee0 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.586395 2224 raft_consensus.cc:2267] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.586833 2224 ts_tablet_manager.cc:1905] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 02:51:34.587404 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet fe860d60b3194dc5a2e7a94d39d4b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe860d60b3194dc5a2e7a94d39d4b8dd already in progress: creating tablet
W20250110 02:51:34.587615 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 3510e86314144a2ca012aacf3d78eacc because the tablet was not found. No further retry: Not found: Tablet not found: 3510e86314144a2ca012aacf3d78eacc
W20250110 02:51:34.587783 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 323ce9f6a3bb4e4a8d80ca0b047500a5 because the tablet was not found. No further retry: Not found: Tablet not found: 323ce9f6a3bb4e4a8d80ca0b047500a5
I20250110 02:51:34.588256 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f1782c46a44ed091f4c533226c71f2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
W20250110 02:51:34.588819 709 leader_election.cc:343] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.588886 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f1782c46a44ed091f4c533226c71f2" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:34.589733 711 leader_election.cc:343] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.589872 711 leader_election.cc:304] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
W20250110 02:51:34.589921 994 consensus_peers.cc:487] T 7064f3f16f6144a4b624eff2ad3ccf62 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Couldn't send request to peer 85f9597077934f1a9814d284ed06d13d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:34.590754 3319 raft_consensus.cc:2743] T b4f1782c46a44ed091f4c533226c71f2 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.594738 2352 ts_tablet_manager.cc:1918] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 02:51:34.594857 2352 log.cc:1198] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/e8b2297377bc4e5095826e30523eede7
I20250110 02:51:34.595189 2352 ts_tablet_manager.cc:1939] T e8b2297377bc4e5095826e30523eede7 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:34.599349 1184 tablet_bootstrap.cc:492] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.599565 1184 ts_tablet_manager.cc:1397] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250110 02:51:34.600426 1184 raft_consensus.cc:357] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.600502 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet e8b2297377bc4e5095826e30523eede7 (table default.Table_daad63f955d54180b631be7ea9671a89 [id=c501975e25c34817b2c02a0e538c97a7]) successfully deleted
I20250110 02:51:34.600697 1184 raft_consensus.cc:383] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.600836 1184 raft_consensus.cc:738] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.601140 1184 consensus_queue.cc:260] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.601799 1184 ts_tablet_manager.cc:1428] T f3e808cfa6df40478892465f7bf5a8e5 P 98336b684b41407da8a1949e30cd82e0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 02:51:34.602614 1184 tablet_bootstrap.cc:492] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0: Bootstrap starting.
I20250110 02:51:34.604549 2352 tablet_replica.cc:331] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:34.604799 2352 raft_consensus.cc:2238] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.604990 2352 raft_consensus.cc:2267] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.605535 2352 ts_tablet_manager.cc:1905] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.609153 2224 ts_tablet_manager.cc:1918] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.609321 2224 log.cc:1198] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/f9b379b357194d729108428cbaebd51a
I20250110 02:51:34.609735 2224 ts_tablet_manager.cc:1939] T f9b379b357194d729108428cbaebd51a P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.610190 1184 tablet_bootstrap.cc:654] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.613679 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 447) took 1591 ms. Trace:
I20250110 02:51:34.615538 2224 rpcz_store.cc:276] 0110 02:51:33.022143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.023011 (+ 868us) service_pool.cc:225] Handling call
0110 02:51:34.585885 (+1562874us) ts_tablet_manager.cc:1145] Deleting tablet f9b379b357194d729108428cbaebd51a
0110 02:51:34.585895 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.599757 (+ 13862us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.602346 (+ 2589us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.613660 (+ 11314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562735}
I20250110 02:51:34.617214 1433 catalog_manager.cc:4872] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): tablet f9b379b357194d729108428cbaebd51a (table default.Table_d837a8b44265419fa2de3fb6bd251f68 [id=b420e79587b046eda24f520cec533837]) successfully deleted
I20250110 02:51:34.617256 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 449) took 1593 ms. Trace:
I20250110 02:51:34.617326 2224 rpcz_store.cc:276] 0110 02:51:33.024281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.024349 (+ 68us) service_pool.cc:225] Handling call
0110 02:51:34.617170 (+1592821us) ts_tablet_manager.cc:1145] Deleting tablet 65285bce60fd4015b085deb7442eea6e
0110 02:51:34.617179 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.617242 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592685}
W20250110 02:51:34.617831 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 65285bce60fd4015b085deb7442eea6e because the tablet was not found. No further retry: Not found: Tablet not found: 65285bce60fd4015b085deb7442eea6e
I20250110 02:51:34.617861 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 439) took 1609 ms. Trace:
I20250110 02:51:34.617923 2224 rpcz_store.cc:276] 0110 02:51:33.008575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.008671 (+ 96us) service_pool.cc:225] Handling call
0110 02:51:34.617780 (+1609109us) ts_tablet_manager.cc:1145] Deleting tablet 5de9cf980d9c4117b4ec3ca937cfcf01
0110 02:51:34.617788 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.617848 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564408}
W20250110 02:51:34.618330 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 5de9cf980d9c4117b4ec3ca937cfcf01 because the tablet was not found. No further retry: Not found: Tablet not found: 5de9cf980d9c4117b4ec3ca937cfcf01
I20250110 02:51:34.618387 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 456) took 1552 ms. Trace:
I20250110 02:51:34.618449 2224 rpcz_store.cc:276] 0110 02:51:33.066171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.066224 (+ 53us) service_pool.cc:225] Handling call
0110 02:51:34.618277 (+1552053us) ts_tablet_manager.cc:1145] Deleting tablet b4f1782c46a44ed091f4c533226c71f2
0110 02:51:34.618286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.618357 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551827}
W20250110 02:51:34.623747 996 consensus_peers.cc:487] T be5ac15f4e774b7e854e4c5b4737b591 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be5ac15f4e774b7e854e4c5b4737b591. This is attempt 1: this message will repeat every 5th retry.
I20250110 02:51:34.623970 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 458) took 1553 ms. Trace:
W20250110 02:51:34.624038 1433 catalog_manager.cc:4852] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b4f1782c46a44ed091f4c533226c71f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4f1782c46a44ed091f4c533226c71f2 already in progress: creating tablet
I20250110 02:51:34.624112 2224 rpcz_store.cc:276] 0110 02:51:33.070130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.070175 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:34.623875 (+1553700us) ts_tablet_manager.cc:1145] Deleting tablet d424b4da66184bd19eccefad80621516
0110 02:51:34.623885 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.623944 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1553537}
W20250110 02:51:34.624578 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d424b4da66184bd19eccefad80621516 because the tablet was not found. No further retry: Not found: Tablet not found: d424b4da66184bd19eccefad80621516
I20250110 02:51:34.624632 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 459) took 1554 ms. Trace:
I20250110 02:51:34.624706 2224 rpcz_store.cc:276] 0110 02:51:33.070268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.070432 (+ 164us) service_pool.cc:225] Handling call
0110 02:51:34.624553 (+1554121us) ts_tablet_manager.cc:1145] Deleting tablet a81c031451104d5a93a1672b4b1918c5
0110 02:51:34.624564 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.624618 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1553992}
I20250110 02:51:34.626973 2357 ts_tablet_manager.cc:1918] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:34.627272 2357 log.cc:1198] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/1d634dd0c2044f91b304fe3737330d88
I20250110 02:51:34.627658 2357 ts_tablet_manager.cc:1939] T 1d634dd0c2044f91b304fe3737330d88 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:34.628546 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet 1d634dd0c2044f91b304fe3737330d88 (table default.Table_8a0645a517224b7ea46c9901cb84ef08 [id=fd521d932d924596b82cfcc49d005a86]) successfully deleted
I20250110 02:51:34.630187 2352 ts_tablet_manager.cc:1918] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.625687 1181 tablet_bootstrap.cc:492] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.630553 1181 ts_tablet_manager.cc:1397] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.001s
W20250110 02:51:34.630647 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet a81c031451104d5a93a1672b4b1918c5 because the tablet was not found. No further retry: Not found: Tablet not found: a81c031451104d5a93a1672b4b1918c5
W20250110 02:51:34.631047 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f69d0c201c524b7e95c9d3851b8dd8c2 because the tablet was not found. No further retry: Not found: Tablet not found: f69d0c201c524b7e95c9d3851b8dd8c2
I20250110 02:51:34.631229 1181 raft_consensus.cc:357] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.631438 1181 raft_consensus.cc:383] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.631489 1181 raft_consensus.cc:738] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
W20250110 02:51:34.631568 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet b45b91d084a44c1e8cdf1553967878e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b45b91d084a44c1e8cdf1553967878e2 already in progress: creating tablet
I20250110 02:51:34.631685 1181 consensus_queue.cc:260] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.632175 1181 ts_tablet_manager.cc:1428] T 78d9140c04a14afc96d43f3bc93488e9 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250110 02:51:34.632136 1433 catalog_manager.cc:4839] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet f6636a3b7f2c4e2fb0ef76f9ab25dcfb because the tablet was not found. No further retry: Not found: Tablet not found: f6636a3b7f2c4e2fb0ef76f9ab25dcfb
I20250110 02:51:34.632409 1181 tablet_bootstrap.cc:492] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.634280 1181 tablet_bootstrap.cc:654] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
W20250110 02:51:34.638072 996 consensus_peers.cc:487] T 36c0aaf2050f481daa8cdac4761fe705 P 8899dd1aeb384e4ea4523f5fd225c888 -> Peer 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): Couldn't send request to peer 98336b684b41407da8a1949e30cd82e0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36c0aaf2050f481daa8cdac4761fe705. This is attempt 1: this message will repeat every 5th retry.
W20250110 02:51:34.639360 1433 catalog_manager.cc:4852] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): delete failed for tablet dc6b2654f86e4686b8a3034de02d11bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc6b2654f86e4686b8a3034de02d11bf already in progress: creating tablet
I20250110 02:51:34.639699 2357 tablet_replica.cc:331] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: stopping tablet replica
I20250110 02:51:34.639873 2352 log.cc:1198] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-1/wal/wals/7efa857721c34003ad91912ef2d7b381
I20250110 02:51:34.640157 2357 raft_consensus.cc:2238] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.641577 2352 ts_tablet_manager.cc:1939] T 7efa857721c34003ad91912ef2d7b381 P 85f9597077934f1a9814d284ed06d13d: Deleting consensus metadata
I20250110 02:51:34.641788 2357 raft_consensus.cc:2267] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.642493 2357 ts_tablet_manager.cc:1905] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.643839 1429 catalog_manager.cc:4872] TS 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): tablet 7efa857721c34003ad91912ef2d7b381 (table default.Table_c42eef47c4634ced8741d5d8bbb6b958 [id=b9c889883b764466b00a516ea67f009c]) successfully deleted
I20250110 02:51:34.644690 2352 tablet_replica.cc:331] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: stopping tablet replica
I20250110 02:51:34.650557 2352 raft_consensus.cc:2238] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.643393 3315 raft_consensus.cc:491] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 02:51:34.631696 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 460) took 1560 ms. Trace:
I20250110 02:51:34.650784 3315 raft_consensus.cc:513] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.650838 2224 rpcz_store.cc:276] 0110 02:51:33.070777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.071169 (+ 392us) service_pool.cc:225] Handling call
0110 02:51:34.630677 (+1559508us) ts_tablet_manager.cc:1145] Deleting tablet d00698a0a9ee4e5da69a745f4a3aa8b4
0110 02:51:34.631628 (+ 951us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.631681 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559357}
I20250110 02:51:34.651444 3315 leader_election.cc:290] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221), 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771)
W20250110 02:51:34.651814 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet d00698a0a9ee4e5da69a745f4a3aa8b4 because the tablet was not found. No further retry: Not found: Tablet not found: d00698a0a9ee4e5da69a745f4a3aa8b4
I20250110 02:51:34.651784 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1253c54d63cc47f081e676851305bc4b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8899dd1aeb384e4ea4523f5fd225c888" is_pre_election: true
W20250110 02:51:34.652418 711 leader_election.cc:343] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.654807 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 461) took 1582 ms. Trace:
I20250110 02:51:34.655174 2224 rpcz_store.cc:276] 0110 02:51:33.072291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.072336 (+ 45us) service_pool.cc:225] Handling call
0110 02:51:34.654718 (+1582382us) ts_tablet_manager.cc:1145] Deleting tablet 676f5f457d9e4028bfd0b707604c42f5
0110 02:51:34.654727 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.654789 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582234}
I20250110 02:51:34.651993 915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1253c54d63cc47f081e676851305bc4b" candidate_uuid: "98336b684b41407da8a1949e30cd82e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f9597077934f1a9814d284ed06d13d" is_pre_election: true
I20250110 02:51:34.655764 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 462) took 1582 ms. Trace:
W20250110 02:51:34.655844 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 676f5f457d9e4028bfd0b707604c42f5 because the tablet was not found. No further retry: Not found: Tablet not found: 676f5f457d9e4028bfd0b707604c42f5
W20250110 02:51:34.655948 709 leader_election.cc:343] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85f9597077934f1a9814d284ed06d13d (127.0.104.2:33221): Illegal state: must be running to vote when last-logged opid is not known
I20250110 02:51:34.656132 2224 rpcz_store.cc:276] 0110 02:51:33.072818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.072862 (+ 44us) service_pool.cc:225] Handling call
0110 02:51:34.655670 (+1582808us) ts_tablet_manager.cc:1145] Deleting tablet 24c4042c38de4969889f4821176fc7bb
0110 02:51:34.655680 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.655736 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582636}
I20250110 02:51:34.656179 709 leader_election.cc:304] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98336b684b41407da8a1949e30cd82e0; no voters: 85f9597077934f1a9814d284ed06d13d, 8899dd1aeb384e4ea4523f5fd225c888
I20250110 02:51:34.656416 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 464) took 1582 ms. Trace:
W20250110 02:51:34.656461 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 24c4042c38de4969889f4821176fc7bb because the tablet was not found. No further retry: Not found: Tablet not found: 24c4042c38de4969889f4821176fc7bb
I20250110 02:51:34.656492 2224 rpcz_store.cc:276] 0110 02:51:33.073741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.073779 (+ 38us) service_pool.cc:225] Handling call
0110 02:51:34.656348 (+1582569us) ts_tablet_manager.cc:1145] Deleting tablet 8eb9ead3175140028e584e1ccb4874de
0110 02:51:34.656357 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.656404 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582367}
W20250110 02:51:34.656829 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 8eb9ead3175140028e584e1ccb4874de because the tablet was not found. No further retry: Not found: Tablet not found: 8eb9ead3175140028e584e1ccb4874de
I20250110 02:51:34.656849 3315 raft_consensus.cc:2743] T 1253c54d63cc47f081e676851305bc4b P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 02:51:34.657181 2352 raft_consensus.cc:2267] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.657832 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 465) took 1582 ms. Trace:
I20250110 02:51:34.657930 2224 rpcz_store.cc:276] 0110 02:51:33.075386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.075426 (+ 40us) service_pool.cc:225] Handling call
0110 02:51:34.657740 (+1582314us) ts_tablet_manager.cc:1145] Deleting tablet b3ae68e4f0254ef38aafd5a1f8f88a54
0110 02:51:34.657752 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.657814 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582172}
I20250110 02:51:34.658799 2352 ts_tablet_manager.cc:1905] T eadb829d657346eabcb7d2aa34b33fe2 P 85f9597077934f1a9814d284ed06d13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.658953 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 467) took 1581 ms. Trace:
W20250110 02:51:34.659039 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b3ae68e4f0254ef38aafd5a1f8f88a54 because the tablet was not found. No further retry: Not found: Tablet not found: b3ae68e4f0254ef38aafd5a1f8f88a54
I20250110 02:51:34.659441 2224 rpcz_store.cc:276] 0110 02:51:33.077057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.077103 (+ 46us) service_pool.cc:225] Handling call
0110 02:51:34.658832 (+1581729us) ts_tablet_manager.cc:1145] Deleting tablet aa6057de9da04f2c82b8f25c5b4e2fe4
0110 02:51:34.658842 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.658923 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574150}
W20250110 02:51:34.660143 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet aa6057de9da04f2c82b8f25c5b4e2fe4 because the tablet was not found. No further retry: Not found: Tablet not found: aa6057de9da04f2c82b8f25c5b4e2fe4
I20250110 02:51:34.660207 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 466) took 1583 ms. Trace:
I20250110 02:51:34.660581 2224 rpcz_store.cc:276] 0110 02:51:33.076500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.076564 (+ 64us) service_pool.cc:225] Handling call
0110 02:51:34.660138 (+1583574us) ts_tablet_manager.cc:1145] Deleting tablet f6df074974974d81a83420e7a3a78070
0110 02:51:34.660146 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.660193 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1575284}
I20250110 02:51:34.661303 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 468) took 1577 ms. Trace:
I20250110 02:51:34.661708 2224 rpcz_store.cc:276] 0110 02:51:33.083859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.083916 (+ 57us) service_pool.cc:225] Handling call
0110 02:51:34.661219 (+1577303us) ts_tablet_manager.cc:1145] Deleting tablet eff9b10c419242cf85ef1b708bd5a862
0110 02:51:34.661230 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.661287 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1575982}
I20250110 02:51:34.662333 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 463) took 1589 ms. Trace:
I20250110 02:51:34.662667 2224 rpcz_store.cc:276] 0110 02:51:33.073290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.077626 (+ 4336us) service_pool.cc:225] Handling call
0110 02:51:34.662266 (+1584640us) ts_tablet_manager.cc:1145] Deleting tablet 27a7885a80414f2193400672aa16a911
0110 02:51:34.662274 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.662321 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576986}
I20250110 02:51:34.663259 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 469) took 1579 ms. Trace:
I20250110 02:51:34.663540 2224 rpcz_store.cc:276] 0110 02:51:33.083988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.084829 (+ 841us) service_pool.cc:225] Handling call
0110 02:51:34.663197 (+1578368us) ts_tablet_manager.cc:1145] Deleting tablet 940445840db3441598eba57d3ccb8c9c
0110 02:51:34.663205 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.663247 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577909}
I20250110 02:51:34.663903 2906 consensus_queue.cc:1035] T 4e54ae5d7e864aff974a107f6762f446 P 8899dd1aeb384e4ea4523f5fd225c888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 02:51:34.664096 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet f6df074974974d81a83420e7a3a78070 because the tablet was not found. No further retry: Not found: Tablet not found: f6df074974974d81a83420e7a3a78070
I20250110 02:51:34.664283 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 470) took 1578 ms. Trace:
W20250110 02:51:34.664331 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet eff9b10c419242cf85ef1b708bd5a862 because the tablet was not found. No further retry: Not found: Tablet not found: eff9b10c419242cf85ef1b708bd5a862
I20250110 02:51:34.664358 2224 rpcz_store.cc:276] 0110 02:51:33.085734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.085855 (+ 121us) service_pool.cc:225] Handling call
0110 02:51:34.664176 (+1578321us) ts_tablet_manager.cc:1145] Deleting tablet b2163c343a694f5ca04ac9f7610d25e3
0110 02:51:34.664185 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.664232 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1578184}
W20250110 02:51:34.664508 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 27a7885a80414f2193400672aa16a911 because the tablet was not found. No further retry: Not found: Tablet not found: 27a7885a80414f2193400672aa16a911
W20250110 02:51:34.664677 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet 940445840db3441598eba57d3ccb8c9c because the tablet was not found. No further retry: Not found: Tablet not found: 940445840db3441598eba57d3ccb8c9c
W20250110 02:51:34.664866 1433 catalog_manager.cc:4839] TS 8899dd1aeb384e4ea4523f5fd225c888 (127.0.104.3:35771): delete failed for tablet b2163c343a694f5ca04ac9f7610d25e3 because the tablet was not found. No further retry: Not found: Tablet not found: b2163c343a694f5ca04ac9f7610d25e3
I20250110 02:51:34.665096 2224 tablet_replica.cc:331] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.665297 2224 raft_consensus.cc:2238] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 LEADER]: Raft consensus shutting down.
I20250110 02:51:34.667151 1181 tablet_bootstrap.cc:492] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888: No bootstrap required, opened a new log
I20250110 02:51:34.667387 1181 ts_tablet_manager.cc:1397] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.005s
I20250110 02:51:34.668256 1181 raft_consensus.cc:357] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.668561 1181 raft_consensus.cc:383] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.668704 1181 raft_consensus.cc:738] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8899dd1aeb384e4ea4523f5fd225c888, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.668989 1181 consensus_queue.cc:260] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.671401 1181 ts_tablet_manager.cc:1428] T 82bd4307e5fe4f61b9308005cf3446e1 P 8899dd1aeb384e4ea4523f5fd225c888: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 02:51:34.671700 1181 tablet_bootstrap.cc:492] T 6694671f422c4be7b545304a38b17edd P 8899dd1aeb384e4ea4523f5fd225c888: Bootstrap starting.
I20250110 02:51:34.675171 2224 raft_consensus.cc:2267] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 02:51:34.678030 2224 ts_tablet_manager.cc:1905] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 02:51:34.679283 1184 tablet_bootstrap.cc:492] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0: No bootstrap required, opened a new log
I20250110 02:51:34.679443 1184 ts_tablet_manager.cc:1397] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.008s
I20250110 02:51:34.679587 1181 tablet_bootstrap.cc:654] T 6694671f422c4be7b545304a38b17edd P 8899dd1aeb384e4ea4523f5fd225c888: Neither blocks nor log segments found. Creating new log.
I20250110 02:51:34.680226 1184 raft_consensus.cc:357] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45139 } }
I20250110 02:51:34.680436 1184 raft_consensus.cc:383] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 02:51:34.680534 1184 raft_consensus.cc:738] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98336b684b41407da8a1949e30cd82e0, State: Initialized, Role: FOLLOWER
I20250110 02:51:34.689024 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 04a4e7142e83493b98f71487df3f304d (table default.Table_0317f9a01941445699da40e583bbe394 [id=d2b34f65b8394d1dac33205441041a77]). Expected version 1 got 0
I20250110 02:51:34.689540 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 235ffe7bf29a4cbcb4faef6f3b2931ec (table default.Table_9b156f13a71144d188c338b19ebe5f7e [id=38abb97bc5e34b318c9a9081c1cdc7bd]). Expected version 2 got 1
I20250110 02:51:34.689934 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ab302fedec4f4c03989b5d3932dc8d4c (table default.Table_74173cc3c3b8435f897e1710a0711981 [id=c6fdcf73aa2a471faeaca5a4a8f70258]). Expected version 2 got 0
I20250110 02:51:34.690264 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ae3033595f824affb2d300c3964e1e8e (table default.Table_1e4195192c1b4741bace326bdbff12e6 [id=18aa3be9f1c74db2a786a1a759849433]). Expected version 1 got 0
I20250110 02:51:34.690469 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e27f7ffc50494854ba34e1fbce9bee1b (table default.Table_aee0556087634aedb308add0c4725dae [id=41e43455ee7c45a6960721c33a18b4d8]). Expected version 1 got 0
I20250110 02:51:34.690640 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 84adc1fd3f76497b98a0fcbc52a3120a (table default.Table_ee66a0c06761457ba22cb964ce907675 [id=fd05c3a951c64cbf81dfd168728165f4]). Expected version 1 got 0
I20250110 02:51:34.690857 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 8c7fbdab52014dddbf63cfef69b09b7b (table default.Table_4c7ba976683d45708a4407a272ee2986 [id=da0f9cd532c7495c89ddcfce7a7b5144]). Expected version 1 got 0
I20250110 02:51:34.691100 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e03d53860ab744968693cf96467c3a19 (table default.Table_4ffa32dac1064dc68c031399c84f8f8c [id=2983e0e083914a099f399cf16f1c30ad]). Expected version 2 got 0
I20250110 02:51:34.691246 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4d8f8d0ac49a428fb7cbabb1e27bf34e (table default.Table_52a56e3c3c3a47f0a95b71d69acc52b3 [id=6985bca790e14c79927bac8621fb5d44]). Expected version 2 got 0
I20250110 02:51:34.691431 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet c1608d90ceb64602889b5705ba4ab55d (table default.Table_d62f105412d049af86f21ec909ac07b5 [id=67bb5fdbe9a249f38f4ec24e99ee42fd]). Expected version 1 got 0
I20250110 02:51:34.691815 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet d3858638b6fc48d0bb8b8f97370a791d (table default.Table_2b13b779fcfb4d5892ff5975b2cd98f0 [id=0861a4be8cc1428285ab917dc1a71f2c]). Expected version 1 got 0
I20250110 02:51:34.692735 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 24cf81039d5e4a47b426ea5d2f21acc2 (table default.Table_614d56e4a7544c27aa6f21776b2138f3 [id=0a7a61acdae94ee392948eaa9495dfcb]). Expected version 3 got 0
I20250110 02:51:34.692929 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet ac5dab241cd74f329a57e6f19e95a7b6 (table default.Table_02d55b18a3b94eaf9c8616d5c7ba42ce [id=38e6ddea3b604a84b097de85815029ba]). Expected version 2 got 0
I20250110 02:51:34.693099 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 4c0b39e2a5a4467ba44926c26d2fd1b2 (table default.Table_95e38b56742441e987008a40d45750fc [id=28ef0a56c1054854a22896fb038a2aac]). Expected version 2 got 1
I20250110 02:51:34.693487 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet 9e28d8769f364e10aad54b60fb221f35 (table default.Table_94b5e8f76ac043d3b234eb6f8b625e2f [id=63a01c80c3a74743be61f355d05e6ddd]). Expected version 1 got 0
I20250110 02:51:34.693732 1449 catalog_manager.cc:5618] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139) does not have the latest schema for tablet e9abd7180bf64bb5991be5f67ea42fbf (table default.Table_0105e3f6dea94fa2a6c83ea8b63220b1 [id=f79a4fd1ae924f818626cfe794d61150]). Expected version 1 got 0
I20250110 02:51:34.696381 2357 ts_tablet_manager.cc:1918] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 02:51:34.696513 2357 log.cc:1198] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-0/wal/wals/e55ee565b7f14936a2cb4cbed716c153
I20250110 02:51:34.696837 2357 ts_tablet_manager.cc:1939] T e55ee565b7f14936a2cb4cbed716c153 P 98336b684b41407da8a1949e30cd82e0: Deleting consensus metadata
I20250110 02:51:34.697777 1433 catalog_manager.cc:4872] TS 98336b684b41407da8a1949e30cd82e0 (127.0.104.1:45139): tablet e55ee565b7f14936a2cb4cbed716c153 (table default.Table_1163b9629c7143d8a9aa6252b1c8b658 [id=c6210f8623084a3ab78a9abc2d4f138d]) successfully deleted
I20250110 02:51:34.697899 2224 ts_tablet_manager.cc:1918] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 02:51:34.698022 2224 log.cc:1198] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting WAL directory at /tmp/dist-test-taskadt4RW/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736477464920669-416-0/minicluster-data/ts-2/wal/wals/5ae4877bba834dbb96ec9675e2aaedd0
I20250110 02:51:34.699450 2224 ts_tablet_manager.cc:1939] T 5ae4877bba834dbb96ec9675e2aaedd0 P 8899dd1aeb384e4ea4523f5fd225c888: Deleting consensus metadata
I20250110 02:51:34.700667 2224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37598 (request call id 471) took 1608 ms. Trace:
I20250110 02:51:34.700771 2224 rpcz_store.cc:276] 0110 02:51:33.091700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 02:51:33.091751 (+ 51us) service_pool.cc:225] Handling call
0110 02:51:34.665000 (+1573249us) ts_tablet_manager.cc:1145] Deleting tablet 5ae4877bba834dbb96ec9675e2aaedd0
0110 02:51:34.665011 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 02:51:34.683858 (+ 18847us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.697793 (+ 13935us) tablet_metadata.cc:641] Metadata flushed
0110 02:51:34.700632 (+ 2839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1573115}
I20250110 02:51:34.702124 2224 tablet_replica.cc:331] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888: stopping tablet replica
I20250110 02:51:34.702339 2224 raft_consensus.cc:2238] T c28f191afa2a43608bbd74322bec97a6 P 8899dd1aeb384e4ea4523f5fd225c888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 02:51:34.680765 1184 consensus_queue.cc:260] T 023b333c321a40dcb83a67ec5fb8ff0e P 98336b684b41407da8a1949e30cd82e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8899dd1aeb384e4ea4523f5fd225c888" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35771 } } peers { permanent_uuid: "85f9597077934f1a9814d284ed06d13d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33221 } } peers { permanent_uuid: "98336b684b41407da8a1949e30cd82e0" member_type: VOTER last_known_addr { host: "127.0.104.1"